[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from WebserverCrawlITest
[ RUN ] WebserverCrawlITest.TestAllWebPages/0
2026-06-19T02:48:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:48:28Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260619 02:48:28.310806 229057 external_mini_cluster.cc:1326] Running /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu
/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu
--fs_wal_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/wal
--fs_data_dirs=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/logs
--server_dump_info_path=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.223.176.126:35389
--webserver_interface=127.223.176.126
--webserver_port=0
--builtin_ntp_servers=127.223.176.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.223.176.126:35389,127.223.176.125:39331,127.223.176.124:33067
--webserver_doc_root=/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www with env {}
W20260619 02:48:28.541196 229097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:48:28.541625 229097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:48:28.541990 229097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:48:28.568471 229097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:48:28.568689 229097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:48:28.568846 229097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:48:28.568991 229097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:48:28.600178 229097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.223.176.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/data
--fs_wal_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.223.176.126:35389,127.223.176.125:39331,127.223.176.124:33067
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.223.176.126:35389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/data/info.pb
--webserver_doc_root=/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www
--webserver_interface=127.223.176.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:29:34 UTC on ccycloud.quasar-rdtsjm.root.comops.site
build id 4454741
TSAN enabled
I20260619 02:48:28.601215 229097 env_posix.cc:2256] Not raising this process' open files per process limit of 1024000; it is already as high as it can go
I20260619 02:48:28.602577 229097 file_cache.cc:493] Constructed file cache file cache with capacity 409600
W20260619 02:48:28.624763 229109 instance_detector.cc:116] could not retrieve GCE instance metadata: Network error: curl error: Couldn't resolve host name: Could not resolve host: metadata.google.internal
W20260619 02:48:29.621201 229108 instance_detector.cc:116] could not retrieve Azure instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1000 milliseconds
W20260619 02:48:29.621587 229107 instance_detector.cc:116] could not retrieve AWS instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1001 milliseconds
W20260619 02:48:29.622164 229110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1001 milliseconds
I20260619 02:48:29.622083 229097 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260619 02:48:30.248625 229097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:48:30.253298 229097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:48:30.254875 229097 hybrid_clock.cc:648] HybridClock initialized: now 1781837310254811 us; error 60 us; skew 500 ppm
I20260619 02:48:30.256480 229097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:48:30.262162 229097 webserver.cc:460] Webserver started at http://127.223.176.126:37059/ using document root /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www and password file <none>
I20260619 02:48:30.262840 229097 fs_manager.cc:359] Metadata directory not provided
I20260619 02:48:30.262969 229097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:48:30.263266 229097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:48:30.266817 229097 fs_manager.cc:1063] Generated new instance metadata in path /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/data/instance:
uuid: "e763ec564a4c4bac91a47833cd1108c4"
format_stamp: "Formatted at 2026-06-19 02:48:30 on ccycloud.quasar-rdtsjm.root.comops.site"
I20260619 02:48:30.267606 229097 fs_manager.cc:1063] Generated new instance metadata in path /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/wal/instance:
uuid: "e763ec564a4c4bac91a47833cd1108c4"
format_stamp: "Formatted at 2026-06-19 02:48:30 on ccycloud.quasar-rdtsjm.root.comops.site"
I20260619 02:48:30.273849 229097 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260619 02:48:30.278945 229117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:30.280218 229097 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260619 02:48:30.280421 229097 fs_manager.cc:642] Opened local filesystem: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/data,/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/wal
uuid: "e763ec564a4c4bac91a47833cd1108c4"
format_stamp: "Formatted at 2026-06-19 02:48:30 on ccycloud.quasar-rdtsjm.root.comops.site"
I20260619 02:48:30.280645 229097 fs_report.cc:383] FS layout report
--------------------
wal directory: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/wal
metadata directory: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/wal
1 data directories: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:48:30.308187 229097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:48:30.309303 229097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 65536; it is already as high as it can go
I20260619 02:48:30.309612 229097 kserver.cc:163] Server-wide thread pool size limit: 6553
I20260619 02:48:30.369323 229097 rpc_server.cc:307] RPC server started. Bound to: 127.223.176.126:35389
I20260619 02:48:30.369740 229168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.223.176.126:35389 every 8 connection(s)
I20260619 02:48:30.371573 229097 server_base.cc:1166] Dumped server information to /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/data/info.pb
I20260619 02:48:30.378415 229169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:48:30.380051 229057 external_mini_cluster.cc:1388] Started /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu as pid 229097
I20260619 02:48:30.380760 229057 external_mini_cluster.cc:1402] Reading /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-0/wal/instance
I20260619 02:48:30.385164 229057 external_mini_cluster.cc:1326] Running /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu
/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu
--fs_wal_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/wal
--fs_data_dirs=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/logs
--server_dump_info_path=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.223.176.125:39331
--webserver_interface=127.223.176.125
--webserver_port=0
--builtin_ntp_servers=127.223.176.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.223.176.126:35389,127.223.176.125:39331,127.223.176.124:33067
--webserver_doc_root=/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www with env {}
I20260619 02:48:30.388302 229169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:48:30.414844 229169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } has no permanent_uuid. Determining permanent_uuid...
W20260619 02:48:30.416937 229120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.223.176.125:39331: connect: Connection refused (error 111)
W20260619 02:48:30.421111 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.125:39331: Network error: Client connection negotiation failed: client connection to 127.223.176.125:39331: connect: Connection refused (error 111)
I20260619 02:48:30.470579 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } attempt: 1
W20260619 02:48:30.476763 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.125:39331: Network error: Client connection negotiation failed: client connection to 127.223.176.125:39331: connect: Connection refused (error 111)
I20260619 02:48:30.545290 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } attempt: 2
W20260619 02:48:30.551705 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.125:39331: Network error: Client connection negotiation failed: client connection to 127.223.176.125:39331: connect: Connection refused (error 111)
W20260619 02:48:30.641069 229171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:48:30.641530 229171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:48:30.641899 229171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 02:48:30.643319 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } attempt: 3
W20260619 02:48:30.648249 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.125:39331: Network error: Client connection negotiation failed: client connection to 127.223.176.125:39331: connect: Connection refused (error 111)
W20260619 02:48:30.668773 229171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:48:30.668977 229171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:48:30.669131 229171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:48:30.669278 229171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:48:30.699992 229171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.223.176.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/data
--fs_wal_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.223.176.126:35389,127.223.176.125:39331,127.223.176.124:33067
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.223.176.125:39331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/data/info.pb
--webserver_doc_root=/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www
--webserver_interface=127.223.176.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:29:34 UTC on ccycloud.quasar-rdtsjm.root.comops.site
build id 4454741
TSAN enabled
I20260619 02:48:30.701089 229171 env_posix.cc:2256] Not raising this process' open files per process limit of 1024000; it is already as high as it can go
I20260619 02:48:30.702430 229171 file_cache.cc:493] Constructed file cache file cache with capacity 409600
W20260619 02:48:30.723199 229182 instance_detector.cc:116] could not retrieve GCE instance metadata: Network error: curl error: Couldn't resolve host name: Could not resolve host: metadata.google.internal
I20260619 02:48:30.791883 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } attempt: 4
W20260619 02:48:30.798337 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.125:39331: Network error: Client connection negotiation failed: client connection to 127.223.176.125:39331: connect: Connection refused (error 111)
I20260619 02:48:31.097729 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } attempt: 5
W20260619 02:48:31.102232 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.125:39331: Network error: Client connection negotiation failed: client connection to 127.223.176.125:39331: connect: Connection refused (error 111)
I20260619 02:48:31.649688 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } attempt: 6
W20260619 02:48:31.659798 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.125:39331: Network error: Client connection negotiation failed: client connection to 127.223.176.125:39331: connect: Connection refused (error 111)
W20260619 02:48:31.721156 229181 instance_detector.cc:116] could not retrieve Azure instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1001 milliseconds
I20260619 02:48:31.721640 229171 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260619 02:48:31.721400 229183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1001 milliseconds
W20260619 02:48:31.721766 229180 instance_detector.cc:116] could not retrieve AWS instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1001 milliseconds
I20260619 02:48:32.345016 229171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:48:32.348217 229171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:48:32.349705 229171 hybrid_clock.cc:648] HybridClock initialized: now 1781837312349654 us; error 58 us; skew 500 ppm
I20260619 02:48:32.350658 229171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:48:32.355685 229171 webserver.cc:460] Webserver started at http://127.223.176.125:36867/ using document root /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www and password file <none>
I20260619 02:48:32.356479 229171 fs_manager.cc:359] Metadata directory not provided
I20260619 02:48:32.356634 229171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:48:32.356993 229171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:48:32.361335 229171 fs_manager.cc:1063] Generated new instance metadata in path /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/data/instance:
uuid: "40ccf050f93e469ca3ace5bd33715e56"
format_stamp: "Formatted at 2026-06-19 02:48:32 on ccycloud.quasar-rdtsjm.root.comops.site"
I20260619 02:48:32.362167 229171 fs_manager.cc:1063] Generated new instance metadata in path /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/wal/instance:
uuid: "40ccf050f93e469ca3ace5bd33715e56"
format_stamp: "Formatted at 2026-06-19 02:48:32 on ccycloud.quasar-rdtsjm.root.comops.site"
I20260619 02:48:32.368181 229171 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 02:48:32.373373 229192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:32.374840 229171 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 02:48:32.375072 229171 fs_manager.cc:642] Opened local filesystem: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/data,/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/wal
uuid: "40ccf050f93e469ca3ace5bd33715e56"
format_stamp: "Formatted at 2026-06-19 02:48:32 on ccycloud.quasar-rdtsjm.root.comops.site"
I20260619 02:48:32.375312 229171 fs_report.cc:383] FS layout report
--------------------
wal directory: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/wal
metadata directory: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/wal
1 data directories: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:48:32.405875 229171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:48:32.407074 229171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 65536; it is already as high as it can go
I20260619 02:48:32.407377 229171 kserver.cc:163] Server-wide thread pool size limit: 6553
I20260619 02:48:32.464463 229171 rpc_server.cc:307] RPC server started. Bound to: 127.223.176.125:39331
I20260619 02:48:32.464782 229243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.223.176.125:39331 every 8 connection(s)
I20260619 02:48:32.466658 229171 server_base.cc:1166] Dumped server information to /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/data/info.pb
I20260619 02:48:32.472669 229057 external_mini_cluster.cc:1388] Started /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu as pid 229171
I20260619 02:48:32.473443 229244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:48:32.473340 229057 external_mini_cluster.cc:1402] Reading /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-1/wal/instance
I20260619 02:48:32.478344 229057 external_mini_cluster.cc:1326] Running /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu
/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu
--fs_wal_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/wal
--fs_data_dirs=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/logs
--server_dump_info_path=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.223.176.124:33067
--webserver_interface=127.223.176.124
--webserver_port=0
--builtin_ntp_servers=127.223.176.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.223.176.126:35389,127.223.176.125:39331,127.223.176.124:33067
--webserver_doc_root=/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www with env {}
I20260619 02:48:32.485244 229244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:48:32.508808 229244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:48:32.528934 229244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } has no permanent_uuid. Determining permanent_uuid...
W20260619 02:48:32.532039 229194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
W20260619 02:48:32.537832 229244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:32.587404 229244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 1
W20260619 02:48:32.593914 229244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:32.662415 229244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 2
W20260619 02:48:32.668728 229244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:32.720156 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } attempt: 7
W20260619 02:48:32.733469 229246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:48:32.733911 229246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:48:32.734287 229246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 02:48:32.735224 229169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } has no permanent_uuid. Determining permanent_uuid...
W20260619 02:48:32.742807 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:32.760286 229244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 3
W20260619 02:48:32.761404 229246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:48:32.761616 229246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:48:32.761772 229246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:48:32.761919 229246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:48:32.766484 229244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:32.792953 229246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.223.176.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/data
--fs_wal_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.223.176.126:35389,127.223.176.125:39331,127.223.176.124:33067
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.223.176.124:33067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/data/info.pb
--webserver_doc_root=/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www
--webserver_interface=127.223.176.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:29:34 UTC on ccycloud.quasar-rdtsjm.root.comops.site
build id 4454741
TSAN enabled
I20260619 02:48:32.794040 229246 env_posix.cc:2256] Not raising this process' open files per process limit of 1024000; it is already as high as it can go
I20260619 02:48:32.795553 229246 file_cache.cc:493] Constructed file cache file cache with capacity 409600
I20260619 02:48:32.801240 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 1
W20260619 02:48:32.807631 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
W20260619 02:48:32.817008 229260 instance_detector.cc:116] could not retrieve GCE instance metadata: Network error: curl error: Couldn't resolve host name: Could not resolve host: metadata.google.internal
I20260619 02:48:32.889096 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 2
W20260619 02:48:32.894678 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:32.910063 229244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 4
W20260619 02:48:32.915961 229244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:32.980108 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 3
W20260619 02:48:32.986967 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:33.127590 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 4
W20260619 02:48:33.133370 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:33.215344 229244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 5
W20260619 02:48:33.221501 229244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:33.416929 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 5
W20260619 02:48:33.420531 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:33.769104 229244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 6
W20260619 02:48:33.776732 229244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:34.032327 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 6
W20260619 02:48:34.043789 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:34.866935 229244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 7
W20260619 02:48:34.878749 229244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:35.077158 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 7
W20260619 02:48:35.085212 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
W20260619 02:48:33.823305 229258 instance_detector.cc:116] could not retrieve AWS instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1009 milliseconds
W20260619 02:48:33.824585 229261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1009 milliseconds
I20260619 02:48:33.826323 229246 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260619 02:48:33.826589 229259 instance_detector.cc:116] could not retrieve Azure instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1011 milliseconds
I20260619 02:48:36.647087 229246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:48:36.650995 229246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:48:36.652390 229246 hybrid_clock.cc:648] HybridClock initialized: now 1781837316652315 us; error 63 us; skew 500 ppm
I20260619 02:48:36.653473 229246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:48:36.659106 229246 webserver.cc:460] Webserver started at http://127.223.176.124:33645/ using document root /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www and password file <none>
I20260619 02:48:36.660301 229246 fs_manager.cc:359] Metadata directory not provided
I20260619 02:48:36.660539 229246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:48:36.660975 229246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:48:36.666975 229246 fs_manager.cc:1063] Generated new instance metadata in path /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/data/instance:
uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0"
format_stamp: "Formatted at 2026-06-19 02:48:36 on ccycloud.quasar-rdtsjm.root.comops.site"
I20260619 02:48:36.668130 229246 fs_manager.cc:1063] Generated new instance metadata in path /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/wal/instance:
uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0"
format_stamp: "Formatted at 2026-06-19 02:48:36 on ccycloud.quasar-rdtsjm.root.comops.site"
I20260619 02:48:36.752972 229246 fs_manager.cc:691] Time spent creating directory manager: real 0.084s user 0.008s sys 0.002s
I20260619 02:48:36.761631 229276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:36.763024 229246 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260619 02:48:36.763325 229246 fs_manager.cc:642] Opened local filesystem: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/data,/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/wal
uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0"
format_stamp: "Formatted at 2026-06-19 02:48:36 on ccycloud.quasar-rdtsjm.root.comops.site"
I20260619 02:48:36.763674 229246 fs_report.cc:383] FS layout report
--------------------
wal directory: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/wal
metadata directory: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/wal
1 data directories: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:48:36.875313 229246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:48:36.952580 229246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 65536; it is already as high as it can go
I20260619 02:48:36.954036 229246 kserver.cc:163] Server-wide thread pool size limit: 6553
I20260619 02:48:36.969189 229244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 8
W20260619 02:48:36.974721 229244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:37.158245 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 8
W20260619 02:48:37.164235 229119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20260619 02:48:37.167799 229169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.176.124:33067: Network error: Client connection negotiation failed: client connection to 127.223.176.124:33067: connect: Connection refused (error 111)
I20260619 02:48:37.189007 229246 rpc_server.cc:307] RPC server started. Bound to: 127.223.176.124:33067
I20260619 02:48:37.189064 229331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.223.176.124:33067 every 8 connection(s)
I20260619 02:48:37.191411 229246 server_base.cc:1166] Dumped server information to /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/data/info.pb
I20260619 02:48:37.266975 229057 external_mini_cluster.cc:1388] Started /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu as pid 229246
I20260619 02:48:37.267522 229057 external_mini_cluster.cc:1402] Reading /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1781837308127060-229057-0/minicluster-data/master-2/wal/instance
I20260619 02:48:37.270078 229332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:48:37.283057 229332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:48:37.382453 229332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:48:37.472368 229332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:48:37.500813 229332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0: Bootstrap starting.
I20260619 02:48:37.587729 229332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0: Neither blocks nor log segments found. Creating new log.
I20260619 02:48:37.589948 229332 log.cc:824] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:48:37.594882 229332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0: No bootstrap required, opened a new log
I20260619 02:48:37.697597 229332 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } }
I20260619 02:48:37.698422 229332 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:48:37.698679 229332 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0b00f0835ef430aa8cd6395cf7f8ef0, State: Initialized, Role: FOLLOWER
I20260619 02:48:37.699671 229332 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } }
I20260619 02:48:37.702348 229343 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } } }
I20260619 02:48:37.703047 229343 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:48:37.703925 229332 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:48:37.804620 229354 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:48:37.804935 229354 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 02:48:37.887571 229282 tablet.cc:2369] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:48:37.803670 229246 master_runner.cc:186] Error getting master registration for 127.223.176.126:35389: OK, instance_id { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" instance_seqno: 1781837310352299 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:48:37.909443 229246 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:48:38.095562 229343 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:48:38.096740 229343 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } }
I20260619 02:48:38.100870 229343 leader_election.cc:288] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e763ec564a4c4bac91a47833cd1108c4 (127.223.176.126:35389), 40ccf050f93e469ca3ace5bd33715e56 (127.223.176.125:39331)
W20260619 02:48:38.831555 229354 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:48:38.831979 229354 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 02:48:39.873339 229354 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:48:39.873737 229354 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 02:48:39.969959 229279 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.223.176.125:39331 timed out after 1.834s (SENT)
W20260619 02:48:40.918175 229354 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:48:40.918448 229354 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:48:41.120419 229244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 9
I20260619 02:48:41.227005 229244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56: Bootstrap starting.
I20260619 02:48:41.235965 229244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56: Neither blocks nor log segments found. Creating new log.
I20260619 02:48:41.238377 229244 log.cc:824] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56: Log is configured to *not* fsync() on all Append() calls
I20260619 02:48:41.244016 229244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56: No bootstrap required, opened a new log
I20260619 02:48:41.327885 229169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } attempt: 9
I20260619 02:48:41.349637 229244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } }
I20260619 02:48:41.350472 229244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:48:41.350754 229244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ccf050f93e469ca3ace5bd33715e56, State: Initialized, Role: FOLLOWER
I20260619 02:48:41.351655 229244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } }
I20260619 02:48:41.441134 229364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:48:41.440564 229365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } } }
I20260619 02:48:41.441629 229365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:48:41.441656 229364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } }
I20260619 02:48:41.443998 229244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:48:41.444628 229219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40ccf050f93e469ca3ace5bd33715e56" is_pre_election: true
I20260619 02:48:41.445467 229169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4: Bootstrap starting.
I20260619 02:48:41.445542 229219 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0b00f0835ef430aa8cd6395cf7f8ef0 in term 0.
I20260619 02:48:41.445989 229364 leader_election.cc:288] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e763ec564a4c4bac91a47833cd1108c4 (127.223.176.126:35389), c0b00f0835ef430aa8cd6395cf7f8ef0 (127.223.176.124:33067)
W20260619 02:48:41.446075 229219 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52264 (request call id 1) took 3344 ms (client timeout 1834 ms). Trace:
W20260619 02:48:41.446466 229219 rpcz_store.cc:269] 0619 02:48:38.101566 (+ 0us) service_pool.cc:170] Inserting onto call queue
0619 02:48:38.101755 (+ 189us) service_pool.cc:227] Handling call
0619 02:48:41.445964 (+3344209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260619 02:48:41.448137 229218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40ccf050f93e469ca3ace5bd33715e56" is_pre_election: true
I20260619 02:48:41.448612 229305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40ccf050f93e469ca3ace5bd33715e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" is_pre_election: true
I20260619 02:48:41.449553 229305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40ccf050f93e469ca3ace5bd33715e56 in term 0.
I20260619 02:48:41.449899 229218 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0b00f0835ef430aa8cd6395cf7f8ef0 in term 0.
W20260619 02:48:41.450318 229218 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52264 (request call id 2) took 1474 ms (client timeout 1834 ms). Trace:
W20260619 02:48:41.450595 229218 rpcz_store.cc:269] 0619 02:48:39.975601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0619 02:48:39.975782 (+ 181us) service_pool.cc:227] Handling call
0619 02:48:41.450254 (+1474472us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260619 02:48:41.451558 229169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4: Neither blocks nor log segments found. Creating new log.
I20260619 02:48:41.452198 229194 leader_election.cc:302] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40ccf050f93e469ca3ace5bd33715e56, c0b00f0835ef430aa8cd6395cf7f8ef0; no voters:
I20260619 02:48:41.452623 229279 leader_election.cc:302] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40ccf050f93e469ca3ace5bd33715e56, c0b00f0835ef430aa8cd6395cf7f8ef0; no voters:
I20260619 02:48:41.453034 229364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:48:41.453301 229364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:48:41.453533 229364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:48:41.454176 229169 log.cc:824] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:48:41.455294 229368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:48:41.455626 229368 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:48:41.455888 229368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:48:41.458763 229364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } }
I20260619 02:48:41.538192 229368 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } }
I20260619 02:48:41.540207 229305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40ccf050f93e469ca3ace5bd33715e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0"
I20260619 02:48:41.541267 229169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4: No bootstrap required, opened a new log
I20260619 02:48:41.542136 229305 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 40ccf050f93e469ca3ace5bd33715e56 in current term 1: Already voted for candidate c0b00f0835ef430aa8cd6395cf7f8ef0 in this term.
I20260619 02:48:41.542876 229364 leader_election.cc:288] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [CANDIDATE]: Term 1 election: Requested vote from peers e763ec564a4c4bac91a47833cd1108c4 (127.223.176.126:35389), c0b00f0835ef430aa8cd6395cf7f8ef0 (127.223.176.124:33067)
I20260619 02:48:41.543536 229218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40ccf050f93e469ca3ace5bd33715e56"
I20260619 02:48:41.542634 229368 leader_election.cc:288] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [CANDIDATE]: Term 1 election: Requested vote from peers e763ec564a4c4bac91a47833cd1108c4 (127.223.176.126:35389), 40ccf050f93e469ca3ace5bd33715e56 (127.223.176.125:39331)
I20260619 02:48:41.544481 229218 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c0b00f0835ef430aa8cd6395cf7f8ef0 in current term 1: Already voted for candidate 40ccf050f93e469ca3ace5bd33715e56 in this term.
I20260619 02:48:41.640733 229169 raft_consensus.cc:348] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } }
I20260619 02:48:41.642884 229169 raft_consensus.cc:374] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:48:41.643666 229169 raft_consensus.cc:729] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e763ec564a4c4bac91a47833cd1108c4, State: Initialized, Role: FOLLOWER
I20260619 02:48:41.646852 229169 consensus_queue.cc:260] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } }
I20260619 02:48:41.651300 229380 sys_catalog.cc:455] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } } }
I20260619 02:48:41.652096 229380 sys_catalog.cc:458] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:48:41.653115 229378 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 02:48:41.653194 229169 sys_catalog.cc:564] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:48:41.653425 229378 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:48:41.653316 229142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40ccf050f93e469ca3ace5bd33715e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e763ec564a4c4bac91a47833cd1108c4" is_pre_election: true
I20260619 02:48:41.653568 229144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e763ec564a4c4bac91a47833cd1108c4" is_pre_election: true
I20260619 02:48:41.653744 229140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e763ec564a4c4bac91a47833cd1108c4"
I20260619 02:48:41.654102 229142 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40ccf050f93e469ca3ace5bd33715e56 in term 0.
W20260619 02:48:41.654598 229144 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36796 (request call id 2) took 3551 ms (client timeout 1834 ms). Trace:
I20260619 02:48:41.654565 229143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e763ec564a4c4bac91a47833cd1108c4" is_pre_election: true
I20260619 02:48:41.655061 229143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0b00f0835ef430aa8cd6395cf7f8ef0 in term 0.
W20260619 02:48:41.655411 229143 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36796 (request call id 3) took 1679 ms (client timeout 1834 ms). Trace:
W20260619 02:48:41.654994 229144 rpcz_store.cc:269] 0619 02:48:38.102782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0619 02:48:38.102982 (+ 200us) service_pool.cc:227] Handling call
0619 02:48:41.654403 (+3551421us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260619 02:48:41.655694 229143 rpcz_store.cc:269] 0619 02:48:39.976106 (+ 0us) service_pool.cc:170] Inserting onto call queue
0619 02:48:39.976492 (+ 386us) service_pool.cc:227] Handling call
0619 02:48:41.655360 (+1678868us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260619 02:48:41.656025 229280 leader_election.cc:302] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0b00f0835ef430aa8cd6395cf7f8ef0; no voters: 40ccf050f93e469ca3ace5bd33715e56, e763ec564a4c4bac91a47833cd1108c4
I20260619 02:48:41.653328 229141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40ccf050f93e469ca3ace5bd33715e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e763ec564a4c4bac91a47833cd1108c4"
I20260619 02:48:41.656955 229368 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 02:48:41.657083 229141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:48:41.662065 229141 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40ccf050f93e469ca3ace5bd33715e56 in term 1.
I20260619 02:48:41.663182 229196 leader_election.cc:302] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40ccf050f93e469ca3ace5bd33715e56, e763ec564a4c4bac91a47833cd1108c4; no voters: c0b00f0835ef430aa8cd6395cf7f8ef0
I20260619 02:48:41.664111 229364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:48:41.664590 229364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [term 1 LEADER]: Becoming Leader. State: Replica: 40ccf050f93e469ca3ace5bd33715e56, State: Running, Role: LEADER
I20260619 02:48:41.665205 229364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } }
W20260619 02:48:41.740976 229198 tablet.cc:2369] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:48:41.741398 229123 tablet.cc:2369] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:48:41.755067 229365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ccf050f93e469ca3ace5bd33715e56. Latest consensus state: current_term: 1 leader_uuid: "40ccf050f93e469ca3ace5bd33715e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } } }
I20260619 02:48:41.757139 229365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [sys.catalog]: This master's current role is: LEADER
I20260619 02:48:41.767428 229388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:48:41.650913 229171 master_runner.cc:186] Error getting master registration for 127.223.176.126:35389: OK, instance_id { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" instance_seqno: 1781837310352299 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:48:41.855214 229171 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:48:41.860785 229388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:48:41.866108 229393 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 02:48:41.959544 229354 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:48:41.959827 229354 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:48:41.971081 229141 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [term 1 FOLLOWER]: Refusing update from remote peer 40ccf050f93e469ca3ace5bd33715e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:48:41.972319 229305 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [term 1 FOLLOWER]: Refusing update from remote peer 40ccf050f93e469ca3ace5bd33715e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:48:41.973242 229365 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:48:41.975044 229364 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:48:42.076296 229368 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ccf050f93e469ca3ace5bd33715e56. Latest consensus state: current_term: 1 leader_uuid: "40ccf050f93e469ca3ace5bd33715e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } } }
I20260619 02:48:42.077297 229368 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:48:42.080703 229380 sys_catalog.cc:455] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ccf050f93e469ca3ace5bd33715e56. Latest consensus state: current_term: 1 leader_uuid: "40ccf050f93e469ca3ace5bd33715e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } } }
I20260619 02:48:42.081264 229380 sys_catalog.cc:458] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:48:42.166302 229364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40ccf050f93e469ca3ace5bd33715e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } } }
I20260619 02:48:42.166987 229364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [sys.catalog]: This master's current role is: LEADER
I20260619 02:48:42.167573 229365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40ccf050f93e469ca3ace5bd33715e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } } }
I20260619 02:48:42.168156 229365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56 [sys.catalog]: This master's current role is: LEADER
I20260619 02:48:42.168934 229396 mvcc.cc:205] Tried to move back new op lower bound from 7298405670388322304 to 7298405669546692608. Current Snapshot: MvccSnapshot[applied={T|T < 7298405670388322304}]
I20260619 02:48:42.171905 229368 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40ccf050f93e469ca3ace5bd33715e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } } }
I20260619 02:48:42.172721 229368 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:48:42.172879 229388 catalog_manager.cc:1347] Generated new cluster ID: de7790bd3a2248cb9e74154db5834d54
I20260619 02:48:42.173139 229388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:48:42.179445 229380 sys_catalog.cc:455] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40ccf050f93e469ca3ace5bd33715e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e763ec564a4c4bac91a47833cd1108c4" member_type: VOTER last_known_addr { host: "127.223.176.126" port: 35389 } } peers { permanent_uuid: "40ccf050f93e469ca3ace5bd33715e56" member_type: VOTER last_known_addr { host: "127.223.176.125" port: 39331 } } peers { permanent_uuid: "c0b00f0835ef430aa8cd6395cf7f8ef0" member_type: VOTER last_known_addr { host: "127.223.176.124" port: 33067 } } }
I20260619 02:48:42.179978 229380 sys_catalog.cc:458] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4 [sys.catalog]: This master's current role is: FOLLOWER
F20260619 02:48:42.273916 229057 external_mini_cluster.cc:634] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1781837322 (unix time) try "date -d @1781837322" if you are using GNU date ***
W20260619 02:48:42.276698 229295 rpcz_store.cc:267] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:49258 (request call id 54) took 6 ms (client timeout 6 ms). Trace:
W20260619 02:48:42.280467 229295 rpcz_store.cc:269] 0619 02:48:42.270610 (+ 0us) service_pool.cc:170] Inserting onto call queue
0619 02:48:42.275560 (+ 4950us) service_pool.cc:227] Handling call
0619 02:48:42.276504 (+ 944us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260619 02:48:42.281625 229388 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:48:42.283911 229388 catalog_manager.cc:1504] Loading token signing keys...
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800037ec1) received by PID 229057 (TID 0x7f6562c92980) from PID 229057; stack trace: ***
@ 0x2b0c50 __tsan::CallUserSignalHandler() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b3340 rtl_sigaction() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f6568ca4630 (unknown) at ??:0
@ 0x7f656400b387 __GI_raise at ??:0
@ 0x7f656400ca78 __GI_abort at ??:0
@ 0x2b5097 __interceptor_abort at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f6567036df4 google::LogMessage::SendToLog() at ??:0
@ 0x7f65670379a0 google::LogMessage::Flush() at ??:0
I20260619 02:48:42.585076 229388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40ccf050f93e469ca3ace5bd33715e56: Generated new TSK 0
I20260619 02:48:42.586366 229388 catalog_manager.cc:1514] Initializing in-progress tserver states...
@ 0x7f656703cadb google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6570c84a2d kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f6570c82aab kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x356057 kudu::WebserverCrawlITest_TestAllWebPages_Test::TestBody() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/webserver-crawl-itest.cc:?
@ 0x7f6567b91e90 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f6567b6fa92 testing::Test::Run() at ??:0
@ 0x7f6567b70da9 testing::TestInfo::Run() at ??:0
@ 0x7f6567b71d85 testing::TestSuite::Run() at ??:0
@ 0x7f6567b86875 testing::internal::UnitTestImpl::RunAllTests() at ??:0
I20260619 02:48:42.907137 229393 catalog_manager.cc:1259] Loaded cluster ID: de7790bd3a2248cb9e74154db5834d54
I20260619 02:48:42.907330 229393 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4: loading cluster ID for follower catalog manager: success
@ 0x7f6567b92d50 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
I20260619 02:48:42.913481 229393 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4: acquiring CA information for follower catalog manager: success
I20260619 02:48:42.918229 229393 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e763ec564a4c4bac91a47833cd1108c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
@ 0x7f6567b85e2d testing::UnitTest::Run() at ??:0
@ 0x38c1cc RUN_ALL_TESTS() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/gtest/gtest.h:?
@ 0x38b08d main at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:?
@ 0x7f6563ff7555 __libc_start_main at ??:0
@ 0x2a9419 (unknown) at ??:?
I20260619 02:48:43.118036 229354 catalog_manager.cc:1259] Loaded cluster ID: de7790bd3a2248cb9e74154db5834d54
I20260619 02:48:43.118309 229354 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0: loading cluster ID for follower catalog manager: success
I20260619 02:48:43.125200 229354 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0: acquiring CA information for follower catalog manager: success
I20260619 02:48:43.215235 229354 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c0b00f0835ef430aa8cd6395cf7f8ef0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 02:49:10.494565 229240 debug-util.cc:398] Leaking SignalData structure 0x7b08000bfea0 after lost signal to thread 229175
W20260619 02:49:10.495383 229240 debug-util.cc:398] Leaking SignalData structure 0x7b0800044520 after lost signal to thread 229243