15:11:50.143 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
15:11:50.143 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskJesFA1/test-tmp
15:11:50.144 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
15:11:50.144 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskJesFA1/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
15:11:50.145 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 3
num_tservers: 3
enable_kerberos: false
cluster_root: "/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590"
hms_mode: NONE
mini_kdc_options {
}
principal: "kudu"
mini_oidc_options {
}
}
15:11:50.458 [DEBUG - main] (MiniKuduCluster.java:195) Response:
15:11:50.459 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}
15:11:50.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:50.461745 24384 mini_oidc.cc:150] Starting JWKS server
15:11:50.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:50.464999 24384 webserver.cc:492] Webserver started at http://127.0.0.1:37899/ using document root <none> and password file <none>
15:11:50.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:50.465353 24384 mini_oidc.cc:186] Starting OIDC Discovery server
15:11:50.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:50.466920 24384 webserver.cc:492] Webserver started at http://127.0.0.1:45167/ using document root <none> and password file <none>
15:11:50.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2025-12-14T15:11:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
15:11:50.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2025-12-14T15:11:50Z Disabled control of system clock
15:11:50.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:50.525560 24384 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:11:50.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/wal
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/data
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/logs
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/data/info.pb
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.23.208.62:41847
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.23.208.62
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.23.208.20:46687
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.23.208.62:41847,127.23.208.61:32899,127.23.208.60:46297
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:11:50.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37899/jwks.json with env {}
15:11:50.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:50.813871 24396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:11:50.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:50.814440 24396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:11:50.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:50.814894 24396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:11:50.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:50.845448 24396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:11:50.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:50.845703 24396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:11:50.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:50.845893 24396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:11:50.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:50.846081 24396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:11:50.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:50.846261 24396 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:11:50.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:50.846414 24396 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:37899/jwks.json
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:50.880151 24396 master_runner.cc:386] Master server non-default flags:
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.23.208.20:46687
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/data
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/wal
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.23.208.62:41847,127.23.208.61:32899,127.23.208.60:46297
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.23.208.62:41847
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37899/jwks.json
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:11:50.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/data/info.pb
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.23.208.62
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/logs
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:11:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:50.881418 24396 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:11:50.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:50.883001 24396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:11:50.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:50.893781 24402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:11:52.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:52.297484 24401 debug-util.cc:398] Leaking SignalData structure 0x7b080002ec80 after lost signal to thread 24396
15:11:52.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:52.661058 24401 kernel_stack_watchdog.cc:198] Thread 24396 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
15:11:52.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
15:11:52.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
15:11:52.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
15:11:52.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
15:11:52.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:50.893941 24403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:11:52.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:52.661929 24396 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.768s user 0.563s sys 1.203s
15:11:52.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:52.662896 24396 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.769s user 0.564s sys 1.204s
15:11:52.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:52.664697 24405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:11:52.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:52.666770 24404 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1767 milliseconds
15:11:52.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.666832 24396 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:11:52.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.668053 24396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:11:52.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.670338 24396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:11:52.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.671692 24396 hybrid_clock.cc:648] HybridClock initialized: now 1765725112671648 us; error 43 us; skew 500 ppm
15:11:52.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.672606 24396 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:11:52.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.678476 24396 webserver.cc:492] Webserver started at http://127.23.208.62:40331/ using document root <none> and password file <none>
15:11:52.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.679430 24396 fs_manager.cc:362] Metadata directory not provided
15:11:52.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.679644 24396 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:11:52.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.680078 24396 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
15:11:52.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.684301 24396 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/data/instance:
15:11:52.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8eef9a17bf6e47819c84bba1eecf4ebc"
15:11:52.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:11:52 on dist-test-slave-6vpz"
15:11:52.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.685475 24396 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/wal/instance:
15:11:52.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8eef9a17bf6e47819c84bba1eecf4ebc"
15:11:52.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:11:52 on dist-test-slave-6vpz"
15:11:52.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.693293 24396 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.001s
15:11:52.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.698673 24412 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:11:52.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.699632 24396 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
15:11:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.699949 24396 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/wal
15:11:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8eef9a17bf6e47819c84bba1eecf4ebc"
15:11:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:11:52 on dist-test-slave-6vpz"
15:11:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.700263 24396 fs_report.cc:389] FS layout report
15:11:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:11:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/wal
15:11:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/wal
15:11:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/data/data
15:11:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:11:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:11:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:11:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:11:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:11:52.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:11:52.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:11:52.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:11:52.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:11:52.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:11:52.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:11:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.751821 24396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:11:52.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.753259 24396 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:11:52.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.753705 24396 kserver.cc:163] Server-wide thread pool size limit: 3276
15:11:52.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.821779 24396 rpc_server.cc:307] RPC server started. Bound to: 127.23.208.62:41847
15:11:52.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.821849 24463 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.208.62:41847 every 8 connection(s)
15:11:52.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.824424 24396 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/data/info.pb
15:11:52.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.829381 24384 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 24396
15:11:52.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.829742 24464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
15:11:52.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.829797 24384 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-0/wal/instance
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.832965 24384 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/wal
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/data
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/logs
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/data/info.pb
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.23.208.61:32899
15:11:52.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.23.208.61
15:11:52.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:11:52.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.23.208.20:46687
15:11:52.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:11:52.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:11:52.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:11:52.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:11:52.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.23.208.62:41847,127.23.208.61:32899,127.23.208.60:46297
15:11:52.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:11:52.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37899/jwks.json with env {}
15:11:52.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.842367 24464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } has no permanent_uuid. Determining permanent_uuid...
15:11:52.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.868752 24464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } has no permanent_uuid. Determining permanent_uuid...
15:11:52.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:52.870868 24413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.208.61:32899: connect: Connection refused (error 111)
15:11:52.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:52.874830 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.61:32899: Network error: Client connection negotiation failed: client connection to 127.23.208.61:32899: connect: Connection refused (error 111)
15:11:52.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.924263 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } attempt: 1
15:11:52.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:52.928723 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.61:32899: Network error: Client connection negotiation failed: client connection to 127.23.208.61:32899: connect: Connection refused (error 111)
15:11:52.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:52.997120 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } attempt: 2
15:11:53.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.001184 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.61:32899: Network error: Client connection negotiation failed: client connection to 127.23.208.61:32899: connect: Connection refused (error 111)
15:11:53.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:53.092561 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } attempt: 3
15:11:53.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.097157 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.61:32899: Network error: Client connection negotiation failed: client connection to 127.23.208.61:32899: connect: Connection refused (error 111)
15:11:53.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.129310 24466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:11:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.129940 24466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:11:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.130496 24466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:11:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.160594 24466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:11:53.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.160974 24466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:11:53.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.161355 24466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:11:53.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.161676 24466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:11:53.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.161979 24466 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:11:53.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.162226 24466 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:37899/jwks.json
15:11:53.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:53.197546 24466 master_runner.cc:386] Master server non-default flags:
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.23.208.20:46687
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/data
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/wal
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.23.208.62:41847,127.23.208.61:32899,127.23.208.60:46297
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.23.208.61:32899
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37899/jwks.json
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/data/info.pb
15:11:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.23.208.61
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/logs
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:11:53.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:53.199134 24466 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:11:53.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:53.200744 24466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:11:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.211565 24475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:11:53.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:53.240535 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } attempt: 4
15:11:53.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.247076 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.61:32899: Network error: Client connection negotiation failed: client connection to 127.23.208.61:32899: connect: Connection refused (error 111)
15:11:53.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:53.546577 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } attempt: 5
15:11:53.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.553830 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.61:32899: Network error: Client connection negotiation failed: client connection to 127.23.208.61:32899: connect: Connection refused (error 111)
15:11:54.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.101366 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } attempt: 6
15:11:54.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:54.126307 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.61:32899: Network error: Client connection negotiation failed: client connection to 127.23.208.61:32899: connect: Connection refused (error 111)
15:11:54.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:54.614711 24474 debug-util.cc:398] Leaking SignalData structure 0x7b080002ec80 after lost signal to thread 24466
15:11:54.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:53.212198 24476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:11:54.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:54.901203 24466 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.690s user 0.620s sys 1.063s
15:11:54.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:54.903002 24466 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.692s user 0.621s sys 1.063s
15:11:54.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:54.903050 24477 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1690 milliseconds
15:11:54.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:54.903692 24478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:11:54.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.903684 24466 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:11:54.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.906641 24466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:11:54.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.909029 24466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:11:54.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.910351 24466 hybrid_clock.cc:648] HybridClock initialized: now 1765725114910310 us; error 41 us; skew 500 ppm
15:11:54.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.911149 24466 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:11:54.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.917307 24466 webserver.cc:492] Webserver started at http://127.23.208.61:40811/ using document root <none> and password file <none>
15:11:54.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.918274 24466 fs_manager.cc:362] Metadata directory not provided
15:11:54.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.918501 24466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:11:54.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.918951 24466 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
15:11:54.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.923166 24466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/data/instance:
15:11:54.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "affa9a43c77042408702d6d9d7b215c8"
15:11:54.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:11:54 on dist-test-slave-6vpz"
15:11:54.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.924557 24466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/wal/instance:
15:11:54.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "affa9a43c77042408702d6d9d7b215c8"
15:11:54.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:11:54 on dist-test-slave-6vpz"
15:11:54.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.931227 24466 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
15:11:54.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.936702 24487 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:11:54.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.937667 24466 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
15:11:54.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.937974 24466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/wal
15:11:54.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "affa9a43c77042408702d6d9d7b215c8"
15:11:54.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:11:54 on dist-test-slave-6vpz"
15:11:54.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.938308 24466 fs_report.cc:389] FS layout report
15:11:54.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:11:54.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/wal
15:11:54.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/wal
15:11:54.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/data/data
15:11:54.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:11:54.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:11:54.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:11:54.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:11:54.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:11:54.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:11:54.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:11:54.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:11:54.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:11:54.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:11:54.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.993264 24466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:11:54.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.995550 24466 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:11:54.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:54.996522 24466 kserver.cc:163] Server-wide thread pool size limit: 3276
15:11:55.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.074700 24466 rpc_server.cc:307] RPC server started. Bound to: 127.23.208.61:32899
15:11:55.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.074746 24538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.208.61:32899 every 8 connection(s)
15:11:55.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.077695 24466 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/data/info.pb
15:11:55.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.080070 24384 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 24466
15:11:55.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.080643 24384 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-1/wal/instance
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.083200 24384 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/wal
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/data
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/logs
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/data/info.pb
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:11:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.23.208.60:46297
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.23.208.60
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.23.208.20:46687
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.23.208.62:41847,127.23.208.61:32899,127.23.208.60:46297
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:11:55.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37899/jwks.json with env {}
15:11:55.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.084769 24539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
15:11:55.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.095098 24539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } has no permanent_uuid. Determining permanent_uuid...
15:11:55.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.115854 24539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } has no permanent_uuid. Determining permanent_uuid...
15:11:55.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.130393 24539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } has no permanent_uuid. Determining permanent_uuid...
15:11:55.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.132772 24488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:55.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.137176 24539 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:55.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.186611 24539 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 1
15:11:55.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.186902 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } attempt: 7
15:11:55.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.191483 24539 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:55.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.198335 24464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } has no permanent_uuid. Determining permanent_uuid...
15:11:55.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.203441 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:55.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.259959 24539 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 2
15:11:55.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.261790 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 1
15:11:55.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.264317 24539 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:55.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.265947 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:55.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.347374 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 2
15:11:55.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.351660 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:55.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.355710 24539 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 3
15:11:55.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.359092 24539 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:55.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.397974 24541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:11:55.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.398531 24541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:11:55.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.399039 24541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:11:55.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.430012 24541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:11:55.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.430317 24541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:11:55.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.430519 24541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:11:55.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.430718 24541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:11:55.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.430909 24541 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:11:55.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.431077 24541 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:37899/jwks.json
15:11:55.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.437021 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 3
15:11:55.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.440615 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.466009 24541 master_runner.cc:386] Master server non-default flags:
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.23.208.20:46687
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/data
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/wal
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.23.208.62:41847,127.23.208.61:32899,127.23.208.60:46297
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.23.208.60:46297
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37899/jwks.json
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/data/info.pb
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.23.208.60
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/logs
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:11:55.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:11:55.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.467236 24541 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:11:55.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.468652 24541 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:11:55.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.480058 24553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:11:55.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.502426 24539 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 4
15:11:55.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.506465 24539 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:55.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.580971 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 4
15:11:55.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.585698 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:55.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.806037 24539 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 5
15:11:55.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.810061 24539 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:55.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:55.869143 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 5
15:11:55.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.873390 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:56.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.357678 24539 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 6
15:11:56.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:56.365952 24539 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:56.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.426034 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 6
15:11:56.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:56.433842 24464 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.23.208.60:46297: Network error: Client connection negotiation failed: client connection to 127.23.208.60:46297: connect: Connection refused (error 111)
15:11:56.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:55.481594 24554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:11:56.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:56.647272 24555 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
15:11:56.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:56.648769 24556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:11:56.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.648779 24541 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:11:56.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.650988 24541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:11:56.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.653514 24541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:11:56.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.654865 24541 hybrid_clock.cc:648] HybridClock initialized: now 1765725116654827 us; error 45 us; skew 500 ppm
15:11:56.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.655762 24541 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:11:56.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.661134 24541 webserver.cc:492] Webserver started at http://127.23.208.60:37765/ using document root <none> and password file <none>
15:11:56.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.662002 24541 fs_manager.cc:362] Metadata directory not provided
15:11:56.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.662230 24541 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:11:56.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.662674 24541 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
15:11:56.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.666837 24541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/data/instance:
15:11:56.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "12eb9000aff74b0891e7b7685fc3fda8"
15:11:56.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:11:56 on dist-test-slave-6vpz"
15:11:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.668041 24541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/wal/instance:
15:11:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "12eb9000aff74b0891e7b7685fc3fda8"
15:11:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:11:56 on dist-test-slave-6vpz"
15:11:56.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.675273 24541 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
15:11:56.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.680699 24567 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:11:56.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.681766 24541 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
15:11:56.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.682044 24541 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/wal
15:11:56.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "12eb9000aff74b0891e7b7685fc3fda8"
15:11:56.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:11:56 on dist-test-slave-6vpz"
15:11:56.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.682332 24541 fs_report.cc:389] FS layout report
15:11:56.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:11:56.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/wal
15:11:56.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/wal
15:11:56.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/data/data
15:11:56.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:11:56.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:11:56.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:11:56.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:11:56.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:11:56.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:11:56.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:11:56.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:11:56.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:11:56.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:11:56.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:11:56.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.753386 24541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:11:56.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.754858 24541 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:11:56.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.755350 24541 kserver.cc:163] Server-wide thread pool size limit: 3276
15:11:56.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.826474 24541 rpc_server.cc:307] RPC server started. Bound to: 127.23.208.60:46297
15:11:56.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.826517 24618 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.208.60:46297 every 8 connection(s)
15:11:56.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.830060 24541 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/data/info.pb
15:11:56.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.832059 24384 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 24541
15:11:56.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.832567 24384 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/master-2/wal/instance
15:11:56.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.837950 24619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
15:11:56.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.853711 24619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } has no permanent_uuid. Determining permanent_uuid...
15:11:56.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.876096 24619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } has no permanent_uuid. Determining permanent_uuid...
15:11:56.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.884735 24619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } has no permanent_uuid. Determining permanent_uuid...
15:11:56.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.899446 24619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8: Bootstrap starting.
15:11:56.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.905791 24619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8: Neither blocks nor log segments found. Creating new log.
15:11:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.907692 24619 log.cc:826] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8: Log is configured to *not* fsync() on all Append() calls
15:11:56.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.912335 24619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8: No bootstrap required, opened a new log
15:11:56.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.930998 24619 raft_consensus.cc:359] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } }
15:11:56.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.931623 24619 raft_consensus.cc:385] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
15:11:56.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.931797 24619 raft_consensus.cc:740] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12eb9000aff74b0891e7b7685fc3fda8, State: Initialized, Role: FOLLOWER
15:11:56.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.932499 24619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } }
15:11:56.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.934691 24628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } } }
15:11:56.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.935320 24628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [sys.catalog]: This master's current role is: FOLLOWER
15:11:56.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.936523 24619 sys_catalog.cc:565] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [sys.catalog]: configured and running, proceeding with master startup.
15:11:56.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:56.960932 24639 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
15:11:56.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:56.961252 24639 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
15:11:56.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.957327 24541 master_runner.cc:186] Error getting master registration for 127.23.208.62:41847: OK, instance_id { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" instance_seqno: 1765725112803010 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
15:11:56.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:56.976286 24541 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:11:57.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:57.010367 24573 tablet.cc:2378] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8: Can't schedule compaction. Clean time has not been advanced past its initial value.
15:11:57.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.125111 24628 raft_consensus.cc:493] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
15:11:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.125649 24628 raft_consensus.cc:515] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } }
15:11:57.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.129336 24628 leader_election.cc:290] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eef9a17bf6e47819c84bba1eecf4ebc (127.23.208.62:41847), affa9a43c77042408702d6d9d7b215c8 (127.23.208.61:32899)
15:11:57.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.426584 24539 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 7
15:11:57.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.451063 24539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8: Bootstrap starting.
15:11:57.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.458905 24539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8: Neither blocks nor log segments found. Creating new log.
15:11:57.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.461125 24539 log.cc:826] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8: Log is configured to *not* fsync() on all Append() calls
15:11:57.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.466419 24539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8: No bootstrap required, opened a new log
15:11:57.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.467214 24464 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } attempt: 7
15:11:57.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.492435 24464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc: Bootstrap starting.
15:11:57.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.496893 24539 raft_consensus.cc:359] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } }
15:11:57.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.497807 24539 raft_consensus.cc:385] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
15:11:57.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.498112 24539 raft_consensus.cc:740] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa9a43c77042408702d6d9d7b215c8, State: Initialized, Role: FOLLOWER
15:11:57.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.499176 24539 consensus_queue.cc:260] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } }
15:11:57.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.500442 24464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc: Neither blocks nor log segments found. Creating new log.
15:11:57.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.503114 24464 log.cc:826] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc: Log is configured to *not* fsync() on all Append() calls
15:11:57.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.502571 24645 sys_catalog.cc:455] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } } }
15:11:57.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.503922 24645 sys_catalog.cc:458] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [sys.catalog]: This master's current role is: FOLLOWER
15:11:57.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.505795 24539 sys_catalog.cc:565] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [sys.catalog]: configured and running, proceeding with master startup.
15:11:57.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.505957 24514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12eb9000aff74b0891e7b7685fc3fda8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa9a43c77042408702d6d9d7b215c8" is_pre_election: true
15:11:57.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:57.506500 24493 tablet.cc:2378] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
15:11:57.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.508152 24514 raft_consensus.cc:2468] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12eb9000aff74b0891e7b7685fc3fda8 in term 0.
15:11:57.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.509773 24568 leader_election.cc:304] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12eb9000aff74b0891e7b7685fc3fda8, affa9a43c77042408702d6d9d7b215c8; no voters:
15:11:57.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.510674 24628 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [term 0 FOLLOWER]: Leader pre-election won for term 1
15:11:57.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.510983 24628 raft_consensus.cc:493] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
15:11:57.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.511335 24628 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [term 0 FOLLOWER]: Advancing to term 1
15:11:57.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.514573 24464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc: No bootstrap required, opened a new log
15:11:57.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.522542 24628 raft_consensus.cc:515] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } }
15:11:57.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.526793 24628 leader_election.cc:290] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [CANDIDATE]: Term 1 election: Requested vote from peers 8eef9a17bf6e47819c84bba1eecf4ebc (127.23.208.62:41847), affa9a43c77042408702d6d9d7b215c8 (127.23.208.61:32899)
15:11:57.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.526906 24514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12eb9000aff74b0891e7b7685fc3fda8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa9a43c77042408702d6d9d7b215c8"
15:11:57.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.527472 24514 raft_consensus.cc:3060] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [term 0 FOLLOWER]: Advancing to term 1
15:11:57.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.534590 24514 raft_consensus.cc:2468] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12eb9000aff74b0891e7b7685fc3fda8 in term 1.
15:11:57.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.535704 24568 leader_election.cc:304] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12eb9000aff74b0891e7b7685fc3fda8, affa9a43c77042408702d6d9d7b215c8; no voters:
15:11:57.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.536665 24628 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [term 1 FOLLOWER]: Leader election won for term 1
15:11:57.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.543566 24628 raft_consensus.cc:697] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [term 1 LEADER]: Becoming Leader. State: Replica: 12eb9000aff74b0891e7b7685fc3fda8, State: Running, Role: LEADER
15:11:57.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.545627 24628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } }
15:11:57.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.559036 24464 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } }
15:11:57.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.560343 24464 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
15:11:57.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.560822 24464 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eef9a17bf6e47819c84bba1eecf4ebc, State: Initialized, Role: FOLLOWER
15:11:57.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.562193 24464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } }
15:11:57.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.569608 24649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12eb9000aff74b0891e7b7685fc3fda8. Latest consensus state: current_term: 1 leader_uuid: "12eb9000aff74b0891e7b7685fc3fda8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } } }
15:11:57.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.570302 24649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [sys.catalog]: This master's current role is: LEADER
15:11:57.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.574194 24654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } } }
15:11:57.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.577231 24438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12eb9000aff74b0891e7b7685fc3fda8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc"
15:11:57.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.577031 24464 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [sys.catalog]: configured and running, proceeding with master startup.
15:11:57.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.577219 24439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12eb9000aff74b0891e7b7685fc3fda8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" is_pre_election: true
15:11:57.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.577745 24654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [sys.catalog]: This master's current role is: FOLLOWER
15:11:57.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.577965 24438 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [term 0 FOLLOWER]: Advancing to term 1
15:11:57.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.584489 24656 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
15:11:57.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.586129 24438 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12eb9000aff74b0891e7b7685fc3fda8 in term 1.
15:11:57.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.596163 24656 catalog_manager.cc:1494] Initializing Kudu cluster ID...
15:11:57.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.614213 24665 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
15:11:57.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.634258 24514 raft_consensus.cc:1275] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [term 1 FOLLOWER]: Refusing update from remote peer 12eb9000aff74b0891e7b7685fc3fda8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
15:11:57.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.634505 24438 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [term 1 FOLLOWER]: Refusing update from remote peer 12eb9000aff74b0891e7b7685fc3fda8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
15:11:57.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.636132 24628 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
15:11:57.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.636929 24649 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
15:11:57.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.640938 24673 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
15:11:57.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.619395 24466 master_runner.cc:186] Error getting master registration for 127.23.208.62:41847: OK, instance_id { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" instance_seqno: 1765725112803010 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
15:11:57.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.656244 24466 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:11:57.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.658385 24654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12eb9000aff74b0891e7b7685fc3fda8. Latest consensus state: current_term: 1 leader_uuid: "12eb9000aff74b0891e7b7685fc3fda8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } } }
15:11:57.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.659065 24654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [sys.catalog]: This master's current role is: FOLLOWER
15:11:57.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.663906 24663 mvcc.cc:204] Tried to move back new op lower bound from 7232410081749422080 to 7232410081487753216. Current Snapshot: MvccSnapshot[applied={T|T < 7232410081749422080}]
15:11:57.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.664320 24674 mvcc.cc:204] Tried to move back new op lower bound from 7232410081749422080 to 7232410081487753216. Current Snapshot: MvccSnapshot[applied={T|T < 7232410081749422080}]
15:11:57.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.668267 24654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12eb9000aff74b0891e7b7685fc3fda8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } } }
15:11:57.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.669701 24654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc [sys.catalog]: This master's current role is: FOLLOWER
15:11:57.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.675206 24649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12eb9000aff74b0891e7b7685fc3fda8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } } }
15:11:57.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.675855 24649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [sys.catalog]: This master's current role is: LEADER
15:11:57.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.676909 24645 sys_catalog.cc:455] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12eb9000aff74b0891e7b7685fc3fda8. Latest consensus state: current_term: 1 leader_uuid: "12eb9000aff74b0891e7b7685fc3fda8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } } }
15:11:57.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.677739 24645 sys_catalog.cc:458] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [sys.catalog]: This master's current role is: FOLLOWER
15:11:57.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.679404 24656 catalog_manager.cc:1357] Generated new cluster ID: f1e55a49ffea4595827d9816ed73f764
15:11:57.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.679733 24656 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
15:11:57.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.686704 24649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12eb9000aff74b0891e7b7685fc3fda8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } } }
15:11:57.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.687412 24649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8 [sys.catalog]: This master's current role is: LEADER
15:11:57.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.694510 24645 sys_catalog.cc:455] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12eb9000aff74b0891e7b7685fc3fda8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eef9a17bf6e47819c84bba1eecf4ebc" member_type: VOTER last_known_addr { host: "127.23.208.62" port: 41847 } } peers { permanent_uuid: "affa9a43c77042408702d6d9d7b215c8" member_type: VOTER last_known_addr { host: "127.23.208.61" port: 32899 } } peers { permanent_uuid: "12eb9000aff74b0891e7b7685fc3fda8" member_type: VOTER last_known_addr { host: "127.23.208.60" port: 46297 } } }
15:11:57.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.695215 24645 sys_catalog.cc:458] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8 [sys.catalog]: This master's current role is: FOLLOWER
15:11:57.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.719175 24656 catalog_manager.cc:1380] Generated new certificate authority record
15:11:57.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.721316 24656 catalog_manager.cc:1514] Loading token signing keys...
15:11:57.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.742686 24656 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 12eb9000aff74b0891e7b7685fc3fda8: Generated new TSK 0
15:11:57.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.743997 24656 catalog_manager.cc:1524] Initializing in-progress tserver states...
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:57.774098 24384 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/wal
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/data
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/logs
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/data/info.pb
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:11:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.23.208.1:0
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.23.208.1
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.23.208.1
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.23.208.62:41847,127.23.208.61:32899,127.23.208.60:46297
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.23.208.20:46687
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:11:57.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
15:11:58.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:58.119558 24682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:11:58.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:58.120345 24682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:11:58.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:58.120918 24682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:11:58.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:58.153332 24682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:11:58.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:58.154114 24682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.208.1
15:11:58.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:58.189543 24682 tablet_server_runner.cc:78] Tablet server non-default flags:
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.23.208.20:46687
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/data
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/wal
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.23.208.1:0
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/data/info.pb
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.23.208.1
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.23.208.62:41847,127.23.208.61:32899,127.23.208.60:46297
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.23.208.1
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/logs
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:11:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:11:58.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:58.190757 24682 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:11:58.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:58.192442 24682 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:11:58.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:58.205476 24688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:11:58.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:58.655341 24665 catalog_manager.cc:1269] Loaded cluster ID: f1e55a49ffea4595827d9816ed73f764
15:11:58.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:58.655695 24665 catalog_manager.cc:1562] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8: loading cluster ID for follower catalog manager: success
15:11:58.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:58.662798 24665 catalog_manager.cc:1584] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8: acquiring CA information for follower catalog manager: success
15:11:58.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:58.667884 24673 catalog_manager.cc:1269] Loaded cluster ID: f1e55a49ffea4595827d9816ed73f764
15:11:58.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:58.668295 24673 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc: loading cluster ID for follower catalog manager: success
15:11:58.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:58.670748 24665 catalog_manager.cc:1612] T 00000000000000000000000000000000 P affa9a43c77042408702d6d9d7b215c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:11:58.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:58.678342 24673 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc: acquiring CA information for follower catalog manager: success
15:11:58.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:11:58.706086 24673 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8eef9a17bf6e47819c84bba1eecf4ebc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:12:00.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:59.609333 24687 debug-util.cc:398] Leaking SignalData structure 0x7b080002ec60 after lost signal to thread 24682
15:12:00.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:00.912012 24687 kernel_stack_watchdog.cc:198] Thread 24682 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
15:12:00.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
15:12:00.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
15:12:00.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
15:12:00.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
15:12:00.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:11:58.206744 24689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:00.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:00.914871 24691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:00.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:00.915591 24682 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.710s user 0.523s sys 1.150s
15:12:00.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:00.916013 24682 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.710s user 0.523s sys 1.150s
15:12:00.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:00.917716 24690 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 2711 milliseconds
15:12:00.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.917770 24682 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:12:00.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.919100 24682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:00.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.921512 24682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:00.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.922937 24682 hybrid_clock.cc:648] HybridClock initialized: now 1765725120922904 us; error 37 us; skew 500 ppm
15:12:00.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.923763 24682 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:00.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.930867 24682 webserver.cc:492] Webserver started at http://127.23.208.1:34767/ using document root <none> and password file <none>
15:12:00.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.931771 24682 fs_manager.cc:362] Metadata directory not provided
15:12:00.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.931989 24682 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:00.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.932499 24682 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
15:12:00.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.936900 24682 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/data/instance:
15:12:00.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c2545cf4037d41b0a0789dfebe495fd7"
15:12:00.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:00 on dist-test-slave-6vpz"
15:12:00.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.938133 24682 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/wal/instance:
15:12:00.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c2545cf4037d41b0a0789dfebe495fd7"
15:12:00.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:00 on dist-test-slave-6vpz"
15:12:00.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.945317 24682 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
15:12:00.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.951331 24702 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:00.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.952417 24682 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
15:12:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.952729 24682 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/wal
15:12:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c2545cf4037d41b0a0789dfebe495fd7"
15:12:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:00 on dist-test-slave-6vpz"
15:12:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:00.953038 24682 fs_report.cc:389] FS layout report
15:12:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/wal
15:12:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/wal
15:12:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/data/data
15:12:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:00.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:00.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:00.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:00.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:00.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:00.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:01.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.017673 24682 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:01.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.019270 24682 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:01.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.019712 24682 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:01.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.023201 24682 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
15:12:01.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.032243 24682 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
15:12:01.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.032570 24682 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
15:12:01.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.032853 24682 ts_tablet_manager.cc:616] Registered 0 tablets
15:12:01.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.033041 24682 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
15:12:01.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.271628 24682 rpc_server.cc:307] RPC server started. Bound to: 127.23.208.1:33163
15:12:01.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.271821 24814 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.208.1:33163 every 8 connection(s)
15:12:01.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.274641 24682 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/data/info.pb
15:12:01.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.282708 24384 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 24682
15:12:01.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.284454 24384 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-0/wal/instance
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.322890 24384 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/wal
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/data
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/logs
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/data/info.pb
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.23.208.2:0
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.23.208.2
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.23.208.2
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.23.208.62:41847,127.23.208.61:32899,127.23.208.60:46297
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.23.208.20:46687
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:01.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
15:12:01.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.403580 24819 heartbeater.cc:344] Connected to a master server at 127.23.208.62:41847
15:12:01.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.404148 24819 heartbeater.cc:461] Registering TS with master...
15:12:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.405617 24819 heartbeater.cc:507] Master 127.23.208.62:41847 requested a full tablet report, sending...
15:12:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.412081 24429 ts_manager.cc:194] Registered new tserver with Master: c2545cf4037d41b0a0789dfebe495fd7 (127.23.208.1:33163)
15:12:01.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.414428 24816 heartbeater.cc:344] Connected to a master server at 127.23.208.61:32899
15:12:01.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.414834 24816 heartbeater.cc:461] Registering TS with master...
15:12:01.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.415915 24816 heartbeater.cc:507] Master 127.23.208.61:32899 requested a full tablet report, sending...
15:12:01.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.420138 24504 ts_manager.cc:194] Registered new tserver with Master: c2545cf4037d41b0a0789dfebe495fd7 (127.23.208.1:33163)
15:12:01.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.425781 24815 heartbeater.cc:344] Connected to a master server at 127.23.208.60:46297
15:12:01.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.426066 24815 heartbeater.cc:461] Registering TS with master...
15:12:01.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.426873 24815 heartbeater.cc:507] Master 127.23.208.60:46297 requested a full tablet report, sending...
15:12:01.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.429479 24584 ts_manager.cc:194] Registered new tserver with Master: c2545cf4037d41b0a0789dfebe495fd7 (127.23.208.1:33163)
15:12:01.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.435329 24584 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.23.208.1:39401
15:12:01.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:01.714854 24824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:01.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:01.715502 24824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:01.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:01.716213 24824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:01.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:01.748944 24824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:01.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:01.749845 24824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.208.2
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.784797 24824 tablet_server_runner.cc:78] Tablet server non-default flags:
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.23.208.20:46687
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/data
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/wal
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.23.208.2:0
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/data/info.pb
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.23.208.2
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.23.208.62:41847,127.23.208.61:32899,127.23.208.60:46297
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.23.208.2
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/logs
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:01.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:01.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.786258 24824 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:01.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:01.787796 24824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:01.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:01.800801 24831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:02.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:02.439532 24815 heartbeater.cc:499] Master 127.23.208.60:46297 was elected leader, sending a full tablet report...
15:12:03.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:03.203965 24830 debug-util.cc:398] Leaking SignalData structure 0x7b080002ec60 after lost signal to thread 24824
15:12:03.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:03.649969 24824 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.846s user 0.705s sys 1.053s
15:12:03.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:03.650866 24824 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.847s user 0.706s sys 1.053s
15:12:03.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:01.803893 24832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:03.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:03.651959 24833 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1847 milliseconds
15:12:03.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:03.652384 24834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:03.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.652446 24824 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:12:03.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.655937 24824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:03.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.657933 24824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:03.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.659255 24824 hybrid_clock.cc:648] HybridClock initialized: now 1765725123659228 us; error 37 us; skew 500 ppm
15:12:03.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.659977 24824 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.665879 24824 webserver.cc:492] Webserver started at http://127.23.208.2:42963/ using document root <none> and password file <none>
15:12:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.666720 24824 fs_manager.cc:362] Metadata directory not provided
15:12:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.666896 24824 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.667333 24824 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
15:12:03.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.671442 24824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/data/instance:
15:12:03.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d42fd319d9d472986350c3a346220bc"
15:12:03.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:03 on dist-test-slave-6vpz"
15:12:03.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.672529 24824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/wal/instance:
15:12:03.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d42fd319d9d472986350c3a346220bc"
15:12:03.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:03 on dist-test-slave-6vpz"
15:12:03.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.679140 24824 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
15:12:03.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.684427 24842 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:03.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.685477 24824 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.685739 24824 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/wal
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d42fd319d9d472986350c3a346220bc"
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:03 on dist-test-slave-6vpz"
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.686007 24824 fs_report.cc:389] FS layout report
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/wal
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/wal
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/data/data
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:03.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.739349 24824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:03.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.740758 24824 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:03.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.741133 24824 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.743624 24824 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
15:12:03.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.747757 24824 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
15:12:03.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.747941 24824 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
15:12:03.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.748142 24824 ts_tablet_manager.cc:616] Registered 0 tablets
15:12:03.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.748265 24824 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
15:12:03.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.890980 24824 rpc_server.cc:307] RPC server started. Bound to: 127.23.208.2:34785
15:12:03.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.891232 24954 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.208.2:34785 every 8 connection(s)
15:12:03.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.893467 24824 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/data/info.pb
15:12:03.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.894718 24384 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 24824
15:12:03.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.895151 24384 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-1/wal/instance
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.906742 24384 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/wal
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/data
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/logs
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/data/info.pb
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:03.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.23.208.3:0
15:12:03.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.23.208.3
15:12:03.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.23.208.3
15:12:03.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:12:03.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.23.208.62:41847,127.23.208.61:32899,127.23.208.60:46297
15:12:03.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.23.208.20:46687
15:12:03.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:03.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:03.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
15:12:03.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.946312 24957 heartbeater.cc:344] Connected to a master server at 127.23.208.62:41847
15:12:03.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.946807 24957 heartbeater.cc:461] Registering TS with master...
15:12:03.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.947448 24956 heartbeater.cc:344] Connected to a master server at 127.23.208.61:32899
15:12:03.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.947633 24955 heartbeater.cc:344] Connected to a master server at 127.23.208.60:46297
15:12:03.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.947775 24956 heartbeater.cc:461] Registering TS with master...
15:12:03.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.947968 24955 heartbeater.cc:461] Registering TS with master...
15:12:03.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.948045 24957 heartbeater.cc:507] Master 127.23.208.62:41847 requested a full tablet report, sending...
15:12:03.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.948779 24956 heartbeater.cc:507] Master 127.23.208.61:32899 requested a full tablet report, sending...
15:12:03.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.948907 24955 heartbeater.cc:507] Master 127.23.208.60:46297 requested a full tablet report, sending...
15:12:03.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.950979 24584 ts_manager.cc:194] Registered new tserver with Master: 3d42fd319d9d472986350c3a346220bc (127.23.208.2:34785)
15:12:03.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.951115 24429 ts_manager.cc:194] Registered new tserver with Master: 3d42fd319d9d472986350c3a346220bc (127.23.208.2:34785)
15:12:03.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.951679 24504 ts_manager.cc:194] Registered new tserver with Master: 3d42fd319d9d472986350c3a346220bc (127.23.208.2:34785)
15:12:03.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:03.952651 24584 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.23.208.2:48411
15:12:04.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:04.229689 24965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:04.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:04.230099 24965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:04.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:04.230530 24965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:04.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:04.260499 24965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:04.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:04.261251 24965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.208.3
15:12:04.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:04.302515 24965 tablet_server_runner.cc:78] Tablet server non-default flags:
15:12:04.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:04.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.23.208.20:46687
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/data
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/wal
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.23.208.3:0
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/data/info.pb
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.23.208.3
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.23.208.62:41847,127.23.208.61:32899,127.23.208.60:46297
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.23.208.3
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/logs
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:04.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:04.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:04.303843 24965 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:04.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:04.305382 24965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:04.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:04.317323 24971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:04.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:04.955341 24955 heartbeater.cc:499] Master 127.23.208.60:46297 was elected leader, sending a full tablet report...
15:12:05.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:04.318424 24972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:05.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:05.559746 24974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:05.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:05.562685 24973 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1239 milliseconds
15:12:05.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.562824 24965 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:12:05.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.564169 24965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:05.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.566721 24965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.568140 24965 hybrid_clock.cc:648] HybridClock initialized: now 1765725125568077 us; error 35 us; skew 500 ppm
15:12:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.568936 24965 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:05.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.575996 24965 webserver.cc:492] Webserver started at http://127.23.208.3:46313/ using document root <none> and password file <none>
15:12:05.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.576970 24965 fs_manager.cc:362] Metadata directory not provided
15:12:05.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.577199 24965 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:05.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.577672 24965 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
15:12:05.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.582055 24965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/data/instance:
15:12:05.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "dc6b4213eda945be9f2591ede9e5b00c"
15:12:05.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:05 on dist-test-slave-6vpz"
15:12:05.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.583300 24965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/wal/instance:
15:12:05.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "dc6b4213eda945be9f2591ede9e5b00c"
15:12:05.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:05 on dist-test-slave-6vpz"
15:12:05.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.590204 24965 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
15:12:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.596081 24982 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.597102 24965 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
15:12:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.597417 24965 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/wal
15:12:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "dc6b4213eda945be9f2591ede9e5b00c"
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:05 on dist-test-slave-6vpz"
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.597750 24965 fs_report.cc:389] FS layout report
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/wal
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/wal
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/data/data
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:05.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.648183 24965 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:05.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.649438 24965 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:05.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.649812 24965 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:05.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.652076 24965 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
15:12:05.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.656034 24965 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
15:12:05.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.656217 24965 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
15:12:05.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.656402 24965 ts_tablet_manager.cc:616] Registered 0 tablets
15:12:05.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.656520 24965 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
15:12:05.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.791595 24965 rpc_server.cc:307] RPC server started. Bound to: 127.23.208.3:46223
15:12:05.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.791692 25094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.208.3:46223 every 8 connection(s)
15:12:05.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.794070 24965 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/data/info.pb
15:12:05.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.800567 24384 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 24965
15:12:05.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.801151 24384 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster5055691034437251590/ts-2/wal/instance
15:12:05.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.849184 25097 heartbeater.cc:344] Connected to a master server at 127.23.208.62:41847
15:12:05.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.849677 25097 heartbeater.cc:461] Registering TS with master...
15:12:05.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.850903 25097 heartbeater.cc:507] Master 127.23.208.62:41847 requested a full tablet report, sending...
15:12:05.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.853782 24429 ts_manager.cc:194] Registered new tserver with Master: dc6b4213eda945be9f2591ede9e5b00c (127.23.208.3:46223)
15:12:05.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.856933 25096 heartbeater.cc:344] Connected to a master server at 127.23.208.61:32899
15:12:05.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.857277 25096 heartbeater.cc:461] Registering TS with master...
15:12:05.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.858072 25096 heartbeater.cc:507] Master 127.23.208.61:32899 requested a full tablet report, sending...
15:12:05.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.860646 24504 ts_manager.cc:194] Registered new tserver with Master: dc6b4213eda945be9f2591ede9e5b00c (127.23.208.3:46223)
15:12:05.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.861250 25095 heartbeater.cc:344] Connected to a master server at 127.23.208.60:46297
15:12:05.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.861596 25095 heartbeater.cc:461] Registering TS with master...
15:12:05.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.862332 25095 heartbeater.cc:507] Master 127.23.208.60:46297 requested a full tablet report, sending...
15:12:05.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.865262 24584 ts_manager.cc:194] Registered new tserver with Master: dc6b4213eda945be9f2591ede9e5b00c (127.23.208.3:46223)
15:12:05.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.867301 24584 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.23.208.3:51589
15:12:05.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:05.872777 24384 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
15:12:05.883 [DEBUG - main] (MiniKuduCluster.java:195) Response:
15:12:05.883 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}
15:12:05.884 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.23.208.62"
port: 41847
}
bound_http_address {
host: "127.23.208.62"
port: 40331
}
}
masters {
id {
type: MASTER
index: 1
}
bound_rpc_address {
host: "127.23.208.61"
port: 32899
}
bound_http_address {
host: "127.23.208.61"
port: 40811
}
}
masters {
id {
type: MASTER
index: 2
}
bound_rpc_address {
host: "127.23.208.60"
port: 46297
}
bound_http_address {
host: "127.23.208.60"
port: 37765
}
}
}
15:12:05.884 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}
15:12:05.886 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
tservers {
id {
type: TSERVER
index: 0
}
bound_rpc_address {
host: "127.23.208.1"
port: 33163
}
bound_http_address {
host: "127.23.208.1"
port: 34767
}
}
tservers {
id {
type: TSERVER
index: 1
}
bound_rpc_address {
host: "127.23.208.2"
port: 34785
}
bound_http_address {
host: "127.23.208.2"
port: 42963
}
}
tservers {
id {
type: TSERVER
index: 2
}
bound_rpc_address {
host: "127.23.208.3"
port: 46223
}
bound_http_address {
host: "127.23.208.3"
port: 46313
}
}
}
15:12:05.886 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
15:12:05.886 [DEBUG - main] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster3313317780826144448/krb5kdc/krb5cc
15:12:05.887 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
15:12:05.888 [INFO - Time-limited test] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskJesFA1/test-tmp
15:12:05.888 [INFO - Time-limited test] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
15:12:05.888 [INFO - Time-limited test] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskJesFA1/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
15:12:05.890 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 3
num_tservers: 0
enable_kerberos: true
cluster_root: "/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687"
extra_master_flags: "--webserver_require_spnego=false"
extra_master_flags: "--authn_token_validity_seconds=10"
extra_tserver_flags: "--webserver_require_spnego=false"
hms_mode: NONE
mini_kdc_options {
ticket_lifetime: "10s"
renew_lifetime: "20s"
}
principal: "kudu"
mini_oidc_options {
}
}
15:12:06.217 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:06.217 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_cluster {
}
15:12:06.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:06.220193 25106 mini_oidc.cc:150] Starting JWKS server
15:12:06.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:06.223438 25106 webserver.cc:492] Webserver started at http://127.0.0.1:45445/ using document root <none> and password file <none>
15:12:06.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:06.223881 25106 mini_oidc.cc:186] Starting OIDC Discovery server
15:12:06.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:06.225615 25106 webserver.cc:492] Webserver started at http://127.0.0.1:36139/ using document root <none> and password file <none>
15:12:06.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Loading random data
15:12:06.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Initializing database '/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/principal' for realm 'KRBTEST.COM',
15:12:06.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master key name 'K/M@KRBTEST.COM'
15:12:06.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:06 dist-test-slave-6vpz krb5kdc[25117](info): setting up network...
15:12:06.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
15:12:06.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:06 dist-test-slave-6vpz krb5kdc[25117](info): set up 2 sockets
15:12:06.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:06 dist-test-slave-6vpz krb5kdc[25117](info): commencing operation
15:12:06.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: starting...
15:12:06.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:06.871141 25095 heartbeater.cc:499] Master 127.23.208.60:46297 was elected leader, sending a full tablet report...
15:12:08.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.267361 25106 mini_kdc.cc:121] Time spent starting KDC: real 2.006s user 0.003s sys 0.015s
15:12:08.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
15:12:08.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
15:12:08.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-admin@KRBTEST.COM" created.
15:12:08.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
15:12:08.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
15:12:08.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-user@KRBTEST.COM" created.
15:12:08.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
15:12:08.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
15:12:08.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "joe-interloper@KRBTEST.COM" created.
15:12:08.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
15:12:08.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/test-user.keytab.
15:12:08.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/test-user.keytab.
15:12:08.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:08 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:08.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
15:12:08.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2025-12-14T15:12:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
15:12:08.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2025-12-14T15:12:08Z Disabled control of system clock
15:12:08.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.24.132.190@KRBTEST.COM; defaulting to no policy
15:12:08.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:12:08.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.24.132.190@KRBTEST.COM" created.
15:12:08.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:12:08.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.24.132.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab.
15:12:08.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.24.132.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab.
15:12:08.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for HTTP/127.24.132.190@KRBTEST.COM; defaulting to no policy
15:12:08.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:12:08.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "HTTP/127.24.132.190@KRBTEST.COM" created.
15:12:08.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:12:08.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal HTTP/127.24.132.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab.
15:12:08.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal HTTP/127.24.132.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab.
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:08.475862 25106 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/logs
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/info.pb
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.190:36349
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.190
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.190
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
15:12:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:12:08.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.765683 25134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:08.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.766350 25134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:08.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.767050 25134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:08.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.798372 25134 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
15:12:08.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.798609 25134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:12:08.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.798974 25134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:08.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.799268 25134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:12:08.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.799463 25134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:12:08.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.799674 25134 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:12:08.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.799837 25134 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45445/jwks.json
15:12:08.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:08.848405 25134 master_runner.cc:386] Master server non-default flags:
15:12:08.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.190
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.190:36349
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/info.pb
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.190
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/logs
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:08.850075 25134 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:08.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:08.851445 25134 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.864454 25140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:10.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.865473 25141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:10.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:08.868922 25143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:10.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.067339 25142 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
15:12:10.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.067425 25134 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:12:10.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.071054 25134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:10.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.074157 25134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:10.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.075583 25134 hybrid_clock.cc:648] HybridClock initialized: now 1765725130075524 us; error 76 us; skew 500 ppm
15:12:10.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.076426 25134 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:10.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:10 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725130, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:10.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.082505 25134 init.cc:377] Logged in from keytab as kudu/127.24.132.190@KRBTEST.COM (short username kudu)
15:12:10.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.088508 25134 webserver.cc:492] Webserver started at http://127.24.132.190:45571/ using document root <none> and password file <none>
15:12:10.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.089346 25134 fs_manager.cc:362] Metadata directory not provided
15:12:10.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.089526 25134 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:10.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.089880 25134 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
15:12:10.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.094085 25134 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/instance:
15:12:10.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "53c07e78e2ea4944abad82dc6a1d4931"
15:12:10.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:10 on dist-test-slave-6vpz"
15:12:10.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.095180 25134 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal/instance:
15:12:10.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "53c07e78e2ea4944abad82dc6a1d4931"
15:12:10.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:10 on dist-test-slave-6vpz"
15:12:10.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.102499 25134 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
15:12:10.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.107714 25152 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:10.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.108776 25134 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.109076 25134 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "53c07e78e2ea4944abad82dc6a1d4931"
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:10 on dist-test-slave-6vpz"
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.109378 25134 fs_report.cc:389] FS layout report
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/data
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:10.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:10.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:10.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.175047 25134 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:10.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.185004 25134 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:10.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.185514 25134 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:10.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.253955 25134 rpc_server.cc:307] RPC server started. Bound to: 127.24.132.190:36349
15:12:10.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.254007 25203 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.132.190:36349 every 8 connection(s)
15:12:10.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.256767 25134 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/info.pb
15:12:10.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.262373 25204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
15:12:10.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.264137 25106 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 25134
15:12:10.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.264595 25106 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal/instance
15:12:10.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.275889 25204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
15:12:10.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.24.132.189@KRBTEST.COM; defaulting to no policy
15:12:10.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:12:10.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.24.132.189@KRBTEST.COM" created.
15:12:10.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:10 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725130, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.190@KRBTEST.COM for kudu/127.24.132.190@KRBTEST.COM
15:12:10.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:12:10.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.24.132.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab.
15:12:10.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.24.132.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab.
15:12:10.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.311281 25204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } has no permanent_uuid. Determining permanent_uuid...
15:12:10.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.313653 25154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:10.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for HTTP/127.24.132.189@KRBTEST.COM; defaulting to no policy
15:12:10.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:12:10.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "HTTP/127.24.132.189@KRBTEST.COM" created.
15:12:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.317749 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.189:40653: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:10.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:12:10.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal HTTP/127.24.132.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab.
15:12:10.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal HTTP/127.24.132.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab.
15:12:10.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.336656 25106 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/logs
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/info.pb
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.189:40653
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.189
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.189
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
15:12:10.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:10.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:10.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:10.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:10.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:10.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:10.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:10.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:10.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
15:12:10.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:12:10.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.367189 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } attempt: 1
15:12:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.370863 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.189:40653: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:10.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.439345 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } attempt: 2
15:12:10.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.443075 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.189:40653: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:10.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.534523 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } attempt: 3
15:12:10.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.538422 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.189:40653: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:10.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.631747 25213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:10.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.632337 25213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:10.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.632973 25213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.664844 25213 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
15:12:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.665063 25213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:12:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.665395 25213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.665611 25213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:12:10.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.665825 25213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:12:10.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.666085 25213 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:12:10.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.666298 25213 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45445/jwks.json
15:12:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.681857 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } attempt: 4
15:12:10.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.685853 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.189:40653: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.703779 25213 master_runner.cc:386] Master server non-default flags:
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.189
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.189:40653
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/info.pb
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.189
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/logs
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:10.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:10.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.704967 25213 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:10.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.705915 25213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:10.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.716452 25220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:10.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:10.985198 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } attempt: 5
15:12:10.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.991667 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.189:40653: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:11.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:11.539322 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } attempt: 6
15:12:11.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:11.549394 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.189:40653: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:11.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.719959 25223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:11.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:10.716971 25221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:11.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:11.987514 25222 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1269 milliseconds
15:12:11.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:11.988991 25213 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:12:11.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:11.992172 25213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:11.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:11.994524 25213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:11.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:11.995827 25213 hybrid_clock.cc:648] HybridClock initialized: now 1765725131995806 us; error 38 us; skew 500 ppm
15:12:11.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:11.996608 25213 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:12.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:11 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725131, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:12.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.001981 25213 init.cc:377] Logged in from keytab as kudu/127.24.132.189@KRBTEST.COM (short username kudu)
15:12:12.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.006866 25213 webserver.cc:492] Webserver started at http://127.24.132.189:34225/ using document root <none> and password file <none>
15:12:12.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.007685 25213 fs_manager.cc:362] Metadata directory not provided
15:12:12.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.007851 25213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:12.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.008195 25213 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
15:12:12.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.012411 25213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/instance:
15:12:12.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0"
15:12:12.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:12 on dist-test-slave-6vpz"
15:12:12.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.013412 25213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal/instance:
15:12:12.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0"
15:12:12.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:12 on dist-test-slave-6vpz"
15:12:12.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.020392 25213 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
15:12:12.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.025621 25235 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:12.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.026672 25213 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
15:12:12.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.026929 25213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:12.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0"
15:12:12.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:12 on dist-test-slave-6vpz"
15:12:12.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.027276 25213 fs_report.cc:389] FS layout report
15:12:12.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:12.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:12.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:12.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/data
15:12:12.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:12.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:12.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:12.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:12.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:12.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:12.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:12.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:12.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:12.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:12.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:12.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.098727 25213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:12.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.103857 25213 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:12.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.104233 25213 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:12.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.174788 25213 rpc_server.cc:307] RPC server started. Bound to: 127.24.132.189:40653
15:12:12.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.174865 25286 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.132.189:40653 every 8 connection(s)
15:12:12.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.177313 25213 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/info.pb
15:12:12.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.182910 25287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
15:12:12.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.183648 25106 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 25213
15:12:12.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.183979 25106 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal/instance
15:12:12.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.196192 25287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
15:12:12.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.24.132.188@KRBTEST.COM; defaulting to no policy
15:12:12.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:12:12.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.24.132.188@KRBTEST.COM" created.
15:12:12.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:12 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725131, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.189@KRBTEST.COM for kudu/127.24.132.190@KRBTEST.COM
15:12:12.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:12:12.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.24.132.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab.
15:12:12.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.24.132.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab.
15:12:12.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.233269 25287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } has no permanent_uuid. Determining permanent_uuid...
15:12:12.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for HTTP/127.24.132.188@KRBTEST.COM; defaulting to no policy
15:12:12.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:12:12.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "HTTP/127.24.132.188@KRBTEST.COM" created.
15:12:12.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:12 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725131, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.189@KRBTEST.COM for kudu/127.24.132.189@KRBTEST.COM
15:12:12.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.261572 25287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } has no permanent_uuid. Determining permanent_uuid...
15:12:12.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:12:12.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal HTTP/127.24.132.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab.
15:12:12.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal HTTP/127.24.132.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab.
15:12:12.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.264191 25238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.265743 25106 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/logs
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/info.pb
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.188:45315
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.188
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.188
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
15:12:12.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:12:12.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.271526 25287 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:12.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.321007 25287 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 1
15:12:12.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.324965 25287 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:12.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.393294 25287 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 2
15:12:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.396996 25287 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:12.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.488350 25287 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 3
15:12:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.492178 25287 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:12.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.570137 25297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:12.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.570639 25297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:12.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.571375 25297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:12.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.602661 25297 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
15:12:12.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.602875 25297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:12:12.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.603206 25297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:12.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.603442 25297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:12:12.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.603638 25297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:12:12.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.603826 25297 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:12:12.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.603969 25297 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45445/jwks.json
15:12:12.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.609930 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } attempt: 7
15:12:12.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:12 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725130, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.190@KRBTEST.COM for kudu/127.24.132.189@KRBTEST.COM
15:12:12.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.632579 25204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } has no permanent_uuid. Determining permanent_uuid...
15:12:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.635528 25287 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 4
15:12:12.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.638197 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:12.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.639483 25287 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.649418 25297 master_runner.cc:386] Master server non-default flags:
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.188
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.188:45315
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/info.pb
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.188
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/logs
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.650548 25297 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.651486 25297 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:12.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.661573 25305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:12.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.696579 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 1
15:12:12.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.703739 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:12.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.785324 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 2
15:12:12.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.802663 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.889156 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 3
15:12:12.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.904421 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:12.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:12.938894 25287 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 5
15:12:12.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.946204 25287 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:13.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:13.044983 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 4
15:12:13.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:13.052423 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:13.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:13.335981 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 5
15:12:13.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:13.344667 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:13.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:13.493772 25287 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 6
15:12:13.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:13.531077 25287 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:13.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:13.897197 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 6
15:12:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:13.913552 25204 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.132.188:45315: Network error: Client connection negotiation failed: client connection to 127.24.132.188:45315: connect: Connection refused (error 111)
15:12:14.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.065654 25304 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ca0 after lost signal to thread 25297
15:12:14.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.219480 25297 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.558s user 0.433s sys 0.978s
15:12:14.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:12.662837 25306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:14.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.219952 25297 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.558s user 0.433s sys 0.978s
15:12:14.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.220736 25307 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1558 milliseconds
15:12:14.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.221812 25308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:14.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.221758 25297 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:12:14.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.226914 25297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:14.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.229390 25297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:14.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.230718 25297 hybrid_clock.cc:648] HybridClock initialized: now 1765725134230692 us; error 32 us; skew 500 ppm
15:12:14.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.231539 25297 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:14.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:14 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725134, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:14.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.238301 25297 init.cc:377] Logged in from keytab as kudu/127.24.132.188@KRBTEST.COM (short username kudu)
15:12:14.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.243456 25297 webserver.cc:492] Webserver started at http://127.24.132.188:41693/ using document root <none> and password file <none>
15:12:14.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.244277 25297 fs_manager.cc:362] Metadata directory not provided
15:12:14.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.244485 25297 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:14.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.244930 25297 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
15:12:14.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.248924 25297 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/instance:
15:12:14.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d4a2215e94e24ae4b512a89aa50d00fa"
15:12:14.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:14 on dist-test-slave-6vpz"
15:12:14.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.249923 25297 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal/instance:
15:12:14.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d4a2215e94e24ae4b512a89aa50d00fa"
15:12:14.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:14 on dist-test-slave-6vpz"
15:12:14.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.257089 25297 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
15:12:14.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.262332 25322 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:14.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.263325 25297 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
15:12:14.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.263624 25297 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:14.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d4a2215e94e24ae4b512a89aa50d00fa"
15:12:14.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:14 on dist-test-slave-6vpz"
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.263916 25297 fs_report.cc:389] FS layout report
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/data
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:14.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:14.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.311345 25297 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:14.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.316664 25297 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:14.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.317175 25297 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:14.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.387876 25297 rpc_server.cc:307] RPC server started. Bound to: 127.24.132.188:45315
15:12:14.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.387957 25373 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.132.188:45315 every 8 connection(s)
15:12:14.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.390499 25297 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/info.pb
15:12:14.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.396487 25374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
15:12:14.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.399381 25106 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 25297
15:12:14.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.399834 25106 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal/instance
15:12:14.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.410871 25374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
15:12:14.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:14 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.132.188@KRBTEST.COM
15:12:14.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:14 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725134, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.188@KRBTEST.COM for kudu/127.24.132.190@KRBTEST.COM
15:12:14.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.442953 25374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } has no permanent_uuid. Determining permanent_uuid...
15:12:14.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:14 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725134, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.188@KRBTEST.COM for kudu/127.24.132.189@KRBTEST.COM
15:12:14.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:14 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.132.189@KRBTEST.COM
15:12:14.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.464882 25374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } has no permanent_uuid. Determining permanent_uuid...
15:12:14.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:14 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725134, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.188@KRBTEST.COM for kudu/127.24.132.188@KRBTEST.COM
15:12:14.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:14 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.132.190@KRBTEST.COM
15:12:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.493676 25374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Bootstrap starting.
15:12:14.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.496821 25106 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
15:12:14.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.499984 25374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Neither blocks nor log segments found. Creating new log.
15:12:14.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.501505 25374 log.cc:826] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Log is configured to *not* fsync() on all Append() calls
15:12:14.503 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:14.504 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_kdc_env_vars {
}
15:12:14.505 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: get_kdc_env_vars {
env_vars {
key: "KRB5CCNAME"
value: "/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc"
}
env_vars {
key: "KRB5_CONFIG"
value: "/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5.conf"
}
env_vars {
key: "KRB5_KDC_PROFILE"
value: "/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kdc.conf"
}
env_vars {
key: "KUDU_ENABLE_KRB5_REALM_FIX"
value: "yes"
}
}
15:12:14.505 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_masters {
}
15:12:14.506 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.24.132.190"
port: 36349
}
bound_http_address {
host: "127.24.132.190"
port: 45571
}
}
masters {
id {
type: MASTER
index: 1
}
bound_rpc_address {
host: "127.24.132.189"
port: 40653
}
bound_http_address {
host: "127.24.132.189"
port: 34225
}
}
masters {
id {
type: MASTER
index: 2
}
bound_rpc_address {
host: "127.24.132.188"
port: 45315
}
bound_http_address {
host: "127.24.132.188"
port: 41693
}
}
}
15:12:14.506 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_tservers {
}
15:12:14.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.506646 25374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: No bootstrap required, opened a new log
15:12:14.507 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:14.507 [INFO - Time-limited test] (MiniKuduCluster.java:622) Running kinit for user test-admin
15:12:14.507 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: kinit {
username: "test-admin"
}
15:12:14.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.532102 25374 raft_consensus.cc:359] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.533046 25374 raft_consensus.cc:385] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
15:12:14.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.533391 25374 raft_consensus.cc:740] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a2215e94e24ae4b512a89aa50d00fa, State: Initialized, Role: FOLLOWER
15:12:14.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:14 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:14.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
15:12:14.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.534286 25374 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.537117 25385 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.537940 25385 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: This master's current role is: FOLLOWER
15:12:14.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.538980 25374 sys_catalog.cc:565] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: configured and running, proceeding with master startup.
15:12:14.539 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:14.540 [DEBUG - Time-limited test] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc
15:12:14.541 [DEBUG - Time-limited test] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
15:12:14.542 [INFO - Time-limited test] (MiniKuduCluster.java:622) Running kinit for user test-user
15:12:14.542 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: kinit {
username: "test-user"
}
15:12:14.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.564340 25397 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
15:12:14.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.564594 25397 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
15:12:14.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.572122 25328 tablet.cc:2378] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
15:12:14.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:14 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:14.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-user@KRBTEST.COM:
15:12:14.579 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:14.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.559465 25297 master_runner.cc:186] Error getting master registration for 127.24.132.190:36349: OK, instance_id { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" instance_seqno: 1765725130235270 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
15:12:14.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.589640 25297 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:12:14.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.591519 25287 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 7
15:12:14.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:14 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725131, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.189@KRBTEST.COM for kudu/127.24.132.188@KRBTEST.COM
15:12:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.622637 25287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Bootstrap starting.
15:12:14.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.630120 25287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Neither blocks nor log segments found. Creating new log.
15:12:14.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.632421 25287 log.cc:826] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Log is configured to *not* fsync() on all Append() calls
15:12:14.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.637287 25287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: No bootstrap required, opened a new log
15:12:14.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.663478 25287 raft_consensus.cc:359] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.664309 25287 raft_consensus.cc:385] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
15:12:14.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.664616 25287 raft_consensus.cc:740] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfea2b69ea4d4c4a9a6837eb55774ae0, State: Initialized, Role: FOLLOWER
15:12:14.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.665508 25287 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.668221 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.668767 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: This master's current role is: FOLLOWER
15:12:14.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.670257 25287 sys_catalog.cc:565] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: configured and running, proceeding with master startup.
15:12:14.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.698362 25413 catalog_manager.cc:1568] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
15:12:14.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.698673 25413 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
15:12:14.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.692229 25213 master_runner.cc:186] Error getting master registration for 127.24.132.190:36349: OK, instance_id { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" instance_seqno: 1765725130235270 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
15:12:14.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.714720 25213 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:12:14.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.799407 25385 raft_consensus.cc:493] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
15:12:14.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.799933 25385 raft_consensus.cc:515] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.802333 25385 leader_election.cc:290] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c07e78e2ea4944abad82dc6a1d4931 (127.24.132.190:36349), cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653)
15:12:14.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.804334 25262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" is_pre_election: true
15:12:14.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.805236 25262 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a2215e94e24ae4b512a89aa50d00fa in term 0.
15:12:14.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.806581 25324 leader_election.cc:304] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfea2b69ea4d4c4a9a6837eb55774ae0, d4a2215e94e24ae4b512a89aa50d00fa; no voters:
15:12:14.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.807458 25385 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Leader pre-election won for term 1
15:12:14.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.807737 25385 raft_consensus.cc:493] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
15:12:14.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.808019 25385 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Advancing to term 1
15:12:14.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.815591 25385 raft_consensus.cc:515] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.818835 25385 leader_election.cc:290] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 1 election: Requested vote from peers 53c07e78e2ea4944abad82dc6a1d4931 (127.24.132.190:36349), cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653)
15:12:14.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.819681 25262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0"
15:12:14.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.820192 25262 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Advancing to term 1
15:12:14.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.829993 25262 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a2215e94e24ae4b512a89aa50d00fa in term 1.
15:12:14.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.831763 25324 leader_election.cc:304] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfea2b69ea4d4c4a9a6837eb55774ae0, d4a2215e94e24ae4b512a89aa50d00fa; no voters:
15:12:14.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.832597 25385 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Leader election won for term 1
15:12:14.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.855141 25385 raft_consensus.cc:697] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 LEADER]: Becoming Leader. State: Replica: d4a2215e94e24ae4b512a89aa50d00fa, State: Running, Role: LEADER
15:12:14.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.856331 25385 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.861999 25241 tablet.cc:2378] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Can't schedule compaction. Clean time has not been advanced past its initial value.
15:12:14.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.878578 25415 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a2215e94e24ae4b512a89aa50d00fa. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.879137 25415 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: This master's current role is: LEADER
15:12:14.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.893849 25417 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
15:12:14.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.908516 25417 catalog_manager.cc:1494] Initializing Kudu cluster ID...
15:12:14.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.933740 25262 raft_consensus.cc:1275] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 1 FOLLOWER]: Refusing update from remote peer d4a2215e94e24ae4b512a89aa50d00fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
15:12:14.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.935312 25415 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
15:12:14.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.947091 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 7
15:12:14.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.968827 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a2215e94e24ae4b512a89aa50d00fa. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.969736 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: This master's current role is: FOLLOWER
15:12:14.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.977600 25418 mvcc.cc:204] Tried to move back new op lower bound from 7232410152671993856 to 7232410152381349888. Current Snapshot: MvccSnapshot[applied={T|T < 7232410152671993856}]
15:12:14.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.989781 25385 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.991524 25385 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: This master's current role is: LEADER
15:12:14.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.996289 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.996965 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: This master's current role is: FOLLOWER
15:12:15.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.003165 25417 catalog_manager.cc:1357] Generated new cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:15.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.003427 25417 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
15:12:15.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:15 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725130, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.190@KRBTEST.COM for kudu/127.24.132.188@KRBTEST.COM
15:12:15.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.045742 25204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Bootstrap starting.
15:12:15.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.054105 25204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Neither blocks nor log segments found. Creating new log.
15:12:15.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.056912 25204 log.cc:826] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Log is configured to *not* fsync() on all Append() calls
15:12:15.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.071456 25417 catalog_manager.cc:1380] Generated new certificate authority record
15:12:15.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.075678 25417 catalog_manager.cc:1514] Loading token signing keys...
15:12:15.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.083077 25204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: No bootstrap required, opened a new log
15:12:15.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.120469 25417 catalog_manager.cc:6027] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Generated new TSK 0
15:12:15.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.121562 25417 catalog_manager.cc:1524] Initializing in-progress tserver states...
15:12:15.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.140190 25204 raft_consensus.cc:359] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:15.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.141217 25204 raft_consensus.cc:385] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
15:12:15.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.141691 25204 raft_consensus.cc:740] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c07e78e2ea4944abad82dc6a1d4931, State: Initialized, Role: FOLLOWER
15:12:15.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.142678 25204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:15.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.146044 25425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:15.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.147066 25425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: FOLLOWER
15:12:15.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.147935 25204 sys_catalog.cc:565] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: configured and running, proceeding with master startup.
15:12:15.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.148138 25178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c07e78e2ea4944abad82dc6a1d4931"
15:12:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.150911 25425 raft_consensus.cc:493] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
15:12:15.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.148860 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c07e78e2ea4944abad82dc6a1d4931" is_pre_election: true
15:12:15.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.151968 25425 raft_consensus.cc:515] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:15.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.153277 25177 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Advancing to term 1
15:12:15.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.154480 25425 leader_election.cc:290] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653), d4a2215e94e24ae4b512a89aa50d00fa (127.24.132.188:45315)
15:12:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.157774 25262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53c07e78e2ea4944abad82dc6a1d4931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" is_pre_election: true
15:12:15.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.158785 25349 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53c07e78e2ea4944abad82dc6a1d4931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" is_pre_election: true
15:12:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.160447 25155 leader_election.cc:304] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53c07e78e2ea4944abad82dc6a1d4931; no voters: cfea2b69ea4d4c4a9a6837eb55774ae0, d4a2215e94e24ae4b512a89aa50d00fa
15:12:15.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.166872 25177 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Refusing update from remote peer d4a2215e94e24ae4b512a89aa50d00fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
15:12:15.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.167514 25425 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
15:12:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.181799 25385 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
15:12:15.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:15.195878 25158 tablet.cc:2378] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Can't schedule compaction. Clean time has not been advanced past its initial value.
15:12:15.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.250007 25431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:15.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.250753 25431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: FOLLOWER
15:12:15.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.252115 25425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a2215e94e24ae4b512a89aa50d00fa. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:15.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.252805 25425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: FOLLOWER
15:12:15.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.315683 25441 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:15.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.316200 25441 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: loading cluster ID for follower catalog manager: success
15:12:15.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.322973 25441 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: acquiring CA information for follower catalog manager: success
15:12:15.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.328615 25441 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:12:15.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.565369 25385 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:15.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.566084 25385 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: This master's current role is: LEADER
15:12:15.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.704675 25413 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:15.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.705173 25413 catalog_manager.cc:1562] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: loading cluster ID for follower catalog manager: success
15:12:15.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.712523 25413 catalog_manager.cc:1584] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: acquiring CA information for follower catalog manager: success
15:12:15.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.719163 25413 catalog_manager.cc:1612] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:12:17.580 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.24.132.188:45315
15:12:17.581 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 2
}
}
15:12:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:17.581744 25106 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 25297
15:12:17.611 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:17.611 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.24.132.190:36349
15:12:17.611 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 0
}
}
15:12:17.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:17.611770 25106 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 25134
15:12:17.635 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:17.636 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.24.132.189:40653
15:12:17.636 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 1
}
}
15:12:17.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:17.636449 25106 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 25213
15:12:17.663 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:17.663 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.24.132.188:45315
15:12:17.663 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 2
}
}
15:12:17.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:17.665438 25106 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:17.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:17.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/logs
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/info.pb
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.188:45315
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.188
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41693
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.188
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
15:12:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:12:17.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:17.970718 25448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:17.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:17.971246 25448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:17.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:17.971984 25448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:18.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.004331 25448 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
15:12:18.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.004637 25448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005059 25448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005296 25448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005494 25448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005686 25448 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005831 25448 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45445/jwks.json
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:18.047106 25448 master_runner.cc:386] Master server non-default flags:
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.188
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.188:45315
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/info.pb
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.188
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41693
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/logs
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:18.048249 25448 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:18.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:18.049167 25448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:18.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.059994 25455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:19.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.060727 25456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:19.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.363989 25458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:19.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.365658 25448 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.306s user 0.501s sys 0.801s
15:12:19.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.365921 25448 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.306s user 0.501s sys 0.801s
15:12:19.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.370180 25457 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1301 milliseconds
15:12:19.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.370242 25448 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:12:19.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.371592 25448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:19.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.374189 25448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:19.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.375533 25448 hybrid_clock.cc:648] HybridClock initialized: now 1765725139375502 us; error 41 us; skew 500 ppm
15:12:19.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.376398 25448 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:19.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:19 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725139, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:19.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.383208 25448 init.cc:377] Logged in from keytab as kudu/127.24.132.188@KRBTEST.COM (short username kudu)
15:12:19.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.388180 25448 webserver.cc:492] Webserver started at http://127.24.132.188:41693/ using document root <none> and password file <none>
15:12:19.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.389067 25448 fs_manager.cc:362] Metadata directory not provided
15:12:19.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.389293 25448 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:19.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.397918 25448 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.003s
15:12:19.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.402453 25467 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:19.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.403578 25448 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.403873 25448 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d4a2215e94e24ae4b512a89aa50d00fa"
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:14 on dist-test-slave-6vpz"
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.406015 25448 fs_report.cc:389] FS layout report
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/data
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:19.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:19.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:19.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.485404 25448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:19.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.495004 25448 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:19.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.495604 25448 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:19.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.567468 25448 rpc_server.cc:307] RPC server started. Bound to: 127.24.132.188:45315
15:12:19.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.567545 25518 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.132.188:45315 every 8 connection(s)
15:12:19.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.570355 25448 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/info.pb
15:12:19.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.579214 25106 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 25448
15:12:19.579 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:19.580 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.24.132.190:36349
15:12:19.580 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 0
}
}
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.581691 25519 sys_catalog.cc:263] Verifying existing consensus state
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.581756 25106 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/logs
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/info.pb
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.190:36349
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.190
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45571
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.190
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:12:19.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.589529 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Bootstrap starting.
15:12:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.628113 25519 log.cc:826] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Log is configured to *not* fsync() on all Append() calls
15:12:19.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.644322 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
15:12:19.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.644994 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Bootstrap complete.
15:12:19.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.665853 25519 raft_consensus.cc:359] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:19.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.668301 25519 raft_consensus.cc:740] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a2215e94e24ae4b512a89aa50d00fa, State: Initialized, Role: FOLLOWER
15:12:19.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.669142 25519 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:19.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.671970 25523 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:19.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.672787 25523 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: This master's current role is: FOLLOWER
15:12:19.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.673650 25519 sys_catalog.cc:565] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: configured and running, proceeding with master startup.
15:12:19.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.703749 25534 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:19.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.704247 25534 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: loading cluster ID for follower catalog manager: success
15:12:19.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.712159 25534 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: acquiring CA information for follower catalog manager: success
15:12:19.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.718102 25534 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:12:19.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.700312 25448 master_runner.cc:186] Error getting master registration for 127.24.132.190:36349: Network error: Client connection negotiation failed: client connection to 127.24.132.190:36349: connect: Connection refused (error 111),
15:12:19.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.725440 25448 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:12:19.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.914695 25521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:19.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.915274 25521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:19.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.916011 25521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.947438 25521 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
15:12:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.947636 25521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.947866 25521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.948041 25521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.948292 25521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.948482 25521 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.948612 25521 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45445/jwks.json
15:12:19.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.986536 25521 master_runner.cc:386] Master server non-default flags:
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.190
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.190:36349
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/info.pb
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.190
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45571
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/logs
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.987830 25521 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:19.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.988814 25521 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:20.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.999444 25542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:21.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:20.006296 25545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:21.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:20.003314 25543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:21.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:21.414235 25544 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1407 milliseconds
15:12:21.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.414363 25521 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:12:21.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.415743 25521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:21.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.419209 25521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:21.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.420646 25521 hybrid_clock.cc:648] HybridClock initialized: now 1765725141420595 us; error 84 us; skew 500 ppm
15:12:21.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.421492 25521 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:21.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:21 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725141, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:21.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.427660 25521 init.cc:377] Logged in from keytab as kudu/127.24.132.190@KRBTEST.COM (short username kudu)
15:12:21.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.433993 25521 webserver.cc:492] Webserver started at http://127.24.132.190:45571/ using document root <none> and password file <none>
15:12:21.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.435099 25521 fs_manager.cc:362] Metadata directory not provided
15:12:21.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.435405 25521 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:21.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.444226 25521 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.004s
15:12:21.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.450044 25555 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:21.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.451213 25521 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
15:12:21.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.451506 25521 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:21.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "53c07e78e2ea4944abad82dc6a1d4931"
15:12:21.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:10 on dist-test-slave-6vpz"
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.453486 25521 fs_report.cc:389] FS layout report
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/data
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:21.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.479501 25556 raft_consensus.cc:493] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
15:12:21.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.480019 25556 raft_consensus.cc:515] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:21.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.485047 25556 leader_election.cc:290] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53c07e78e2ea4944abad82dc6a1d4931 (127.24.132.190:36349), cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653)
15:12:21.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:21.488711 25471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.132.190:36349: connect: Connection refused (error 111)
15:12:21.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:21.493407 25471 leader_election.cc:336] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 53c07e78e2ea4944abad82dc6a1d4931 (127.24.132.190:36349): Network error: Client connection negotiation failed: client connection to 127.24.132.190:36349: connect: Connection refused (error 111)
15:12:21.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:21.494419 25469 leader_election.cc:336] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653): Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:21.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.494931 25469 leader_election.cc:304] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a2215e94e24ae4b512a89aa50d00fa; no voters: 53c07e78e2ea4944abad82dc6a1d4931, cfea2b69ea4d4c4a9a6837eb55774ae0
15:12:21.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.496363 25556 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
15:12:21.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.513373 25521 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:21.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.519672 25521 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:21.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.520233 25521 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:21.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.613556 25521 rpc_server.cc:307] RPC server started. Bound to: 127.24.132.190:36349
15:12:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.613631 25610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.132.190:36349 every 8 connection(s)
15:12:21.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.616478 25521 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/info.pb
15:12:21.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.625711 25106 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 25521
15:12:21.626 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:21.626 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.24.132.189:40653
15:12:21.626 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 1
}
}
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.628283 25106 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/logs
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/info.pb
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.189:40653
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.189
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34225
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.189
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.629547 25611 sys_catalog.cc:263] Verifying existing consensus state
15:12:21.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.636662 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Bootstrap starting.
15:12:21.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.672662 25611 log.cc:826] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Log is configured to *not* fsync() on all Append() calls
15:12:21.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.687310 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
15:12:21.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.688105 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Bootstrap complete.
15:12:21.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.709476 25611 raft_consensus.cc:359] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:21.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.711678 25611 raft_consensus.cc:740] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c07e78e2ea4944abad82dc6a1d4931, State: Initialized, Role: FOLLOWER
15:12:21.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.712601 25611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:21.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.715384 25615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:21.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.716375 25615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: FOLLOWER
15:12:21.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.717427 25611 sys_catalog.cc:565] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: configured and running, proceeding with master startup.
15:12:21.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.777666 25627 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:21.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.777963 25627 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: loading cluster ID for follower catalog manager: success
15:12:21.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.785459 25627 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: acquiring CA information for follower catalog manager: success
15:12:21.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:21 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725141, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.190@KRBTEST.COM for kudu/127.24.132.190@KRBTEST.COM
15:12:21.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.790601 25627 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:12:21.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.813221 25521 master_runner.cc:186] Error getting master registration for 127.24.132.189:40653: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111),
15:12:21.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.871994 25521 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:12:22.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.070969 25613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:22.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.071935 25613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:22.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.073133 25613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:22.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.134161 25613 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
15:12:22.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.134469 25613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:12:22.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.134922 25613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:22.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.135293 25613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:12:22.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.135634 25613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:12:22.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.136040 25613 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:12:22.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.136310 25613 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45445/jwks.json
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:22.202206 25613 master_runner.cc:386] Master server non-default flags:
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.189
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.189:40653
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/info.pb
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.189
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34225
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/logs
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:22.203781 25613 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:22.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:22.204998 25613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:22.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.218295 25636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:23.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.374578 25642 raft_consensus.cc:493] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
15:12:23.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.375514 25642 raft_consensus.cc:515] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:23.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.393575 25642 leader_election.cc:290] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653), d4a2215e94e24ae4b512a89aa50d00fa (127.24.132.188:45315)
15:12:23.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.394549 25561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:23.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.405519 25561 leader_election.cc:336] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653): Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:23.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.416551 25494 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53c07e78e2ea4944abad82dc6a1d4931" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" is_pre_election: true
15:12:23.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.417461 25494 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53c07e78e2ea4944abad82dc6a1d4931 in term 1.
15:12:23.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.421586 25562 leader_election.cc:304] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53c07e78e2ea4944abad82dc6a1d4931, d4a2215e94e24ae4b512a89aa50d00fa; no voters: cfea2b69ea4d4c4a9a6837eb55774ae0
15:12:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.423069 25642 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Leader pre-election won for term 2
15:12:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.423477 25642 raft_consensus.cc:493] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
15:12:23.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.423846 25642 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Advancing to term 2
15:12:23.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.435166 25642 raft_consensus.cc:515] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:23.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.441318 25561 leader_election.cc:336] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653): Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:23.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.443399 25494 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53c07e78e2ea4944abad82dc6a1d4931" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a2215e94e24ae4b512a89aa50d00fa"
15:12:23.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.443948 25494 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Advancing to term 2
15:12:23.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.454635 25642 leader_election.cc:290] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 election: Requested vote from peers cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653), d4a2215e94e24ae4b512a89aa50d00fa (127.24.132.188:45315)
15:12:23.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.456370 25494 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c07e78e2ea4944abad82dc6a1d4931 in term 2.
15:12:23.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.457604 25562 leader_election.cc:304] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53c07e78e2ea4944abad82dc6a1d4931, d4a2215e94e24ae4b512a89aa50d00fa; no voters: cfea2b69ea4d4c4a9a6837eb55774ae0
15:12:23.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.458644 25642 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 2 FOLLOWER]: Leader election won for term 2
15:12:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.464550 25642 raft_consensus.cc:697] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 2 LEADER]: Becoming Leader. State: Replica: 53c07e78e2ea4944abad82dc6a1d4931, State: Running, Role: LEADER
15:12:23.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.465499 25642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:23.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.480492 25647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c07e78e2ea4944abad82dc6a1d4931. Latest consensus state: current_term: 2 leader_uuid: "53c07e78e2ea4944abad82dc6a1d4931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:23.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.481451 25647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: LEADER
15:12:23.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.494287 25649 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
15:12:23.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.502856 25649 catalog_manager.cc:1494] Initializing Kudu cluster ID...
15:12:23.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.506880 25649 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:23.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.507140 25649 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
15:12:23.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.512351 25649 catalog_manager.cc:1514] Loading token signing keys...
15:12:23.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.520992 25649 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Loaded TSK: 0
15:12:23.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.522305 25649 catalog_manager.cc:1524] Initializing in-progress tserver states...
15:12:23.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.619637 25635 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ca0 after lost signal to thread 25613
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.711859 25635 kernel_stack_watchdog.cc:198] Thread 25613 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 394ms:
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.220979 25637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:23.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.721180 25613 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.498s user 0.002s sys 0.000s
15:12:23.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.721606 25613 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.498s user 0.002s sys 0.000s
15:12:23.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.723213 25639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:23.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.723402 25613 server_base.cc:1047] running on GCE node
15:12:23.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.724716 25613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:23.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.727129 25613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:23.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.728513 25613 hybrid_clock.cc:648] HybridClock initialized: now 1765725143728443 us; error 72 us; skew 500 ppm
15:12:23.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.729368 25613 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:23.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:23 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725143, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:23.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.736028 25613 init.cc:377] Logged in from keytab as kudu/127.24.132.189@KRBTEST.COM (short username kudu)
15:12:23.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.742038 25613 webserver.cc:492] Webserver started at http://127.24.132.189:34225/ using document root <none> and password file <none>
15:12:23.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.743299 25613 fs_manager.cc:362] Metadata directory not provided
15:12:23.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.743592 25613 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:23.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.752440 25613 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
15:12:23.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.757129 25656 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.758571 25613 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
15:12:23.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.758953 25613 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:23.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0"
15:12:23.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:12 on dist-test-slave-6vpz"
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.761613 25613 fs_report.cc:389] FS layout report
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/data
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:23.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.822003 25613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:23.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.829385 25613 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:23.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.829911 25613 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:23.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.923925 25561 consensus_peers.cc:597] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 -> Peer cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653): Couldn't send request to peer cfea2b69ea4d4c4a9a6837eb55774ae0. Status: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
15:12:23.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.952184 25613 rpc_server.cc:307] RPC server started. Bound to: 127.24.132.189:40653
15:12:23.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.952219 25707 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.132.189:40653 every 8 connection(s)
15:12:23.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.955983 25613 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/info.pb
15:12:23.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.961740 25106 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 25613
15:12:23.962 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:23.963 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.24.132.188:45315(127.24.132.188:45315)] connecting to peer
15:12:23.965 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.24.132.190:36349(127.24.132.190:36349)] connecting to peer
15:12:23.967 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.24.132.189:40653(127.24.132.189:40653)] connecting to peer
15:12:23.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.970239 25708 sys_catalog.cc:263] Verifying existing consensus state
15:12:23.971 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.24.132.190:36349(127.24.132.190:36349)] Successfully connected to peer
15:12:23.980 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.24.132.188:45315(127.24.132.188:45315)] Successfully connected to peer
15:12:23.981 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.24.132.189:40653(127.24.132.189:40653)] Successfully connected to peer
15:12:23.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.981603 25708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Bootstrap starting.
15:12:23.985 [DEBUG - kudu-nio-1] (SecurityContext.java:222) Refreshing Kerberos credentials...
15:12:23.986 [DEBUG - kudu-nio-1] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc
15:12:24.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:24 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:24.005 [DEBUG - kudu-nio-1] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
15:12:24.005 [WARN - kudu-nio-1] (SecurityContext.java:231) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
15:12:24.005 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.24.132.190:36349(127.24.132.190:36349)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.006 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.24.132.190:36349: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.006 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.24.132.190:36349(127.24.132.190:36349)] handling channelInactive
15:12:24.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:24.008308 25714 negotiation.cc:337] Failed RPC negotiation. Trace:
15:12:24.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.972322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60060 (local address 127.24.132.190:36349)
15:12:24.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983300 (+ 10978us) server_negotiation.cc:207] Beginning negotiation
15:12:24.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983338 (+ 38us) server_negotiation.cc:400] Waiting for connection header
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983390 (+ 52us) server_negotiation.cc:408] Connection header received
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983532 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983559 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983775 (+ 216us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.001560 (+ 17785us) server_negotiation.cc:227] Negotiated authn=SASL
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.006915 (+ 5355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60060: BlockingRecv error: recv got EOF from 127.0.0.1:60060 (error 108)
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":10330,"thread_start_us":10054,"threads_started":1}
15:12:24.011 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.24.132.188:45315(127.24.132.188:45315)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.012 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.24.132.188:45315: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.012 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.24.132.188:45315(127.24.132.188:45315)] handling channelInactive
15:12:24.013 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.24.132.189:40653(127.24.132.189:40653)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.013 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.24.132.189:40653: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:24.015595 25713 negotiation.cc:337] Failed RPC negotiation. Trace:
15:12:24.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.968097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.24.132.188:45315)
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005607 (+ 37510us) server_negotiation.cc:207] Beginning negotiation
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005639 (+ 32us) server_negotiation.cc:400] Waiting for connection header
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005688 (+ 49us) server_negotiation.cc:408] Connection header received
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005824 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005849 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.006139 (+ 290us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.013562 (+ 7423us) server_negotiation.cc:227] Negotiated authn=SASL
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.014158 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48212: BlockingRecv error: recv got EOF from 127.0.0.1:48212 (error 108)
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":36915,"thread_start_us":38878,"threads_started":1}
15:12:24.015 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1523) Notify RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.188:45315, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.190:36349, [5ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.189:40653, [44ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.190:36349: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [49ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.188:45315: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [51ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.189:40653: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) after lookup exception
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.24.132.188:45315,127.24.132.190:36349,127.24.132.189:40653). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268) [main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) [main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) [main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) [main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) [main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) [main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) [main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) [main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289) [netty-handler-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.24.132.188:45315,127.24.132.190:36349,127.24.132.189:40653). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
15:12:24.019 [DEBUG - kudu-nio-0] (KuduRpc.java:260) Handling a callback on RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=56), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.188:45315, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.190:36349, [5ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.189:40653, [44ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.190:36349: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [49ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.188:45315: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [51ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.189:40653: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) with no deferred attached!
15:12:24.019 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.24.132.189:40653(127.24.132.189:40653)] handling channelInactive
15:12:24.022 [DEBUG - main] (Connection.java:575) [peer master-127.24.132.190:36349(127.24.132.190:36349)] disconnecting while in state TERMINATED
15:12:24.022 [DEBUG - main] (Connection.java:575) [peer master-127.24.132.189:40653(127.24.132.189:40653)] disconnecting while in state TERMINATED
15:12:24.023 [DEBUG - main] (Connection.java:575) [peer master-127.24.132.188:45315(127.24.132.188:45315)] disconnecting while in state TERMINATED
15:12:24.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:24.022956 25715 negotiation.cc:337] Failed RPC negotiation. Trace:
15:12:24.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.980765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.24.132.189:40653)
15:12:24.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.000005 (+ 19240us) server_negotiation.cc:207] Beginning negotiation
15:12:24.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.000057 (+ 52us) server_negotiation.cc:400] Waiting for connection header
15:12:24.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.000124 (+ 67us) server_negotiation.cc:408] Connection header received
15:12:24.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.009901 (+ 9777us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:12:24.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.009992 (+ 91us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:12:24.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.010687 (+ 695us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:12:24.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.011401 (+ 714us) server_negotiation.cc:227] Negotiated authn=SASL
15:12:24.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.020311 (+ 8910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56020: BlockingRecv error: recv got EOF from 127.0.0.1:56020 (error 108)
15:12:24.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":18512,"thread_start_us":18028,"threads_started":1}
15:12:24.024 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2704) Releasing all remaining resources
15:12:24.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:24.031706 25106 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 25521
15:12:24.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:24.040874 25708 log.cc:826] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Log is configured to *not* fsync() on all Append() calls
15:12:24.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:24.057823 25708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
15:12:24.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:24.058619 25708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Bootstrap complete.
15:12:24.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:24.061965 25106 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 25613
15:12:24.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:24.080113 25106 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 25448
15:12:25.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2025-12-14T15:12:25Z chronyd exiting
15:12:26.392 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
15:12:26.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:26.394049 24384 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 24682
15:12:26.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:26.417058 24384 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 24824
15:12:26.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:26.442685 24384 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 24965
15:12:26.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:26.468418 24384 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 24396
15:12:26.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:26.495172 24384 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 24466
15:12:26.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:26.523464 24384 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 24541
15:12:27.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2025-12-14T15:12:27Z chronyd exiting
15:12:28.685 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient: failed attempt 1
java.lang.AssertionError:
Expected: a string containing "found that the new Kerberos principal test-user@KRBTEST.COM did not match the original principal test-admin@KRBTEST.COM"
but: was "15:12:14.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.559465 25297 master_runner.cc:186] Error getting master registration for 127.24.132.190:36349: OK, instance_id { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" instance_seqno: 1765725130235270 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
15:12:14.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.589640 25297 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:12:14.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.591519 25287 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 7
15:12:14.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:14 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725131, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.189@KRBTEST.COM for kudu/127.24.132.188@KRBTEST.COM
15:12:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.622637 25287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Bootstrap starting.
15:12:14.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.630120 25287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Neither blocks nor log segments found. Creating new log.
15:12:14.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.632421 25287 log.cc:826] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Log is configured to *not* fsync() on all Append() calls
15:12:14.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.637287 25287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: No bootstrap required, opened a new log
15:12:14.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.663478 25287 raft_consensus.cc:359] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.664309 25287 raft_consensus.cc:385] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
15:12:14.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.664616 25287 raft_consensus.cc:740] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfea2b69ea4d4c4a9a6837eb55774ae0, State: Initialized, Role: FOLLOWER
15:12:14.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.665508 25287 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.668221 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.668767 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: This master's current role is: FOLLOWER
15:12:14.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.670257 25287 sys_catalog.cc:565] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: configured and running, proceeding with master startup.
15:12:14.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.698362 25413 catalog_manager.cc:1568] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
15:12:14.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.698673 25413 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
15:12:14.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.692229 25213 master_runner.cc:186] Error getting master registration for 127.24.132.190:36349: OK, instance_id { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" instance_seqno: 1765725130235270 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
15:12:14.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.714720 25213 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:12:14.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.799407 25385 raft_consensus.cc:493] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
15:12:14.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.799933 25385 raft_consensus.cc:515] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.802333 25385 leader_election.cc:290] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c07e78e2ea4944abad82dc6a1d4931 (127.24.132.190:36349), cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653)
15:12:14.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.804334 25262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" is_pre_election: true
15:12:14.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.805236 25262 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a2215e94e24ae4b512a89aa50d00fa in term 0.
15:12:14.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.806581 25324 leader_election.cc:304] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfea2b69ea4d4c4a9a6837eb55774ae0, d4a2215e94e24ae4b512a89aa50d00fa; no voters:
15:12:14.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.807458 25385 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Leader pre-election won for term 1
15:12:14.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.807737 25385 raft_consensus.cc:493] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
15:12:14.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.808019 25385 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Advancing to term 1
15:12:14.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.815591 25385 raft_consensus.cc:515] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.818835 25385 leader_election.cc:290] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 1 election: Requested vote from peers 53c07e78e2ea4944abad82dc6a1d4931 (127.24.132.190:36349), cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653)
15:12:14.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.819681 25262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0"
15:12:14.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.820192 25262 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Advancing to term 1
15:12:14.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.829993 25262 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a2215e94e24ae4b512a89aa50d00fa in term 1.
15:12:14.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.831763 25324 leader_election.cc:304] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfea2b69ea4d4c4a9a6837eb55774ae0, d4a2215e94e24ae4b512a89aa50d00fa; no voters:
15:12:14.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.832597 25385 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Leader election won for term 1
15:12:14.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.855141 25385 raft_consensus.cc:697] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 LEADER]: Becoming Leader. State: Replica: d4a2215e94e24ae4b512a89aa50d00fa, State: Running, Role: LEADER
15:12:14.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.856331 25385 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.861999 25241 tablet.cc:2378] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Can't schedule compaction. Clean time has not been advanced past its initial value.
15:12:14.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.878578 25415 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a2215e94e24ae4b512a89aa50d00fa. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.879137 25415 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: This master's current role is: LEADER
15:12:14.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.893849 25417 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
15:12:14.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.908516 25417 catalog_manager.cc:1494] Initializing Kudu cluster ID...
15:12:14.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.933740 25262 raft_consensus.cc:1275] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 1 FOLLOWER]: Refusing update from remote peer d4a2215e94e24ae4b512a89aa50d00fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
15:12:14.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.935312 25415 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
15:12:14.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.947091 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 7
15:12:14.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.968827 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a2215e94e24ae4b512a89aa50d00fa. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.969736 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: This master's current role is: FOLLOWER
15:12:14.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.977600 25418 mvcc.cc:204] Tried to move back new op lower bound from 7232410152671993856 to 7232410152381349888. Current Snapshot: MvccSnapshot[applied={T|T < 7232410152671993856}]
15:12:14.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.989781 25385 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.991524 25385 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: This master's current role is: LEADER
15:12:14.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.996289 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.996965 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: This master's current role is: FOLLOWER
15:12:15.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.003165 25417 catalog_manager.cc:1357] Generated new cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:15.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.003427 25417 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
15:12:15.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:15 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725130, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.190@KRBTEST.COM for kudu/127.24.132.188@KRBTEST.COM
15:12:15.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.045742 25204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Bootstrap starting.
15:12:15.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.054105 25204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Neither blocks nor log segments found. Creating new log.
15:12:15.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.056912 25204 log.cc:826] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Log is configured to *not* fsync() on all Append() calls
15:12:15.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.071456 25417 catalog_manager.cc:1380] Generated new certificate authority record
15:12:15.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.075678 25417 catalog_manager.cc:1514] Loading token signing keys...
15:12:15.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.083077 25204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: No bootstrap required, opened a new log
15:12:15.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.120469 25417 catalog_manager.cc:6027] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Generated new TSK 0
15:12:15.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.121562 25417 catalog_manager.cc:1524] Initializing in-progress tserver states...
15:12:15.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.140190 25204 raft_consensus.cc:359] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:15.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.141217 25204 raft_consensus.cc:385] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
15:12:15.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.141691 25204 raft_consensus.cc:740] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c07e78e2ea4944abad82dc6a1d4931, State: Initialized, Role: FOLLOWER
15:12:15.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.142678 25204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:15.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.146044 25425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:15.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.147066 25425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: FOLLOWER
15:12:15.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.147935 25204 sys_catalog.cc:565] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: configured and running, proceeding with master startup.
15:12:15.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.148138 25178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c07e78e2ea4944abad82dc6a1d4931"
15:12:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.150911 25425 raft_consensus.cc:493] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
15:12:15.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.148860 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c07e78e2ea4944abad82dc6a1d4931" is_pre_election: true
15:12:15.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.151968 25425 raft_consensus.cc:515] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:15.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.153277 25177 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Advancing to term 1
15:12:15.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.154480 25425 leader_election.cc:290] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653), d4a2215e94e24ae4b512a89aa50d00fa (127.24.132.188:45315)
15:12:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.157774 25262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53c07e78e2ea4944abad82dc6a1d4931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" is_pre_election: true
15:12:15.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.158785 25349 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53c07e78e2ea4944abad82dc6a1d4931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" is_pre_election: true
15:12:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.160447 25155 leader_election.cc:304] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53c07e78e2ea4944abad82dc6a1d4931; no voters: cfea2b69ea4d4c4a9a6837eb55774ae0, d4a2215e94e24ae4b512a89aa50d00fa
15:12:15.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.166872 25177 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Refusing update from remote peer d4a2215e94e24ae4b512a89aa50d00fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
15:12:15.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.167514 25425 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
15:12:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.181799 25385 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
15:12:15.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:15.195878 25158 tablet.cc:2378] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Can't schedule compaction. Clean time has not been advanced past its initial value.
15:12:15.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.250007 25431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:15.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.250753 25431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: FOLLOWER
15:12:15.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.252115 25425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a2215e94e24ae4b512a89aa50d00fa. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:15.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.252805 25425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: FOLLOWER
15:12:15.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.315683 25441 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:15.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.316200 25441 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: loading cluster ID for follower catalog manager: success
15:12:15.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.322973 25441 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: acquiring CA information for follower catalog manager: success
15:12:15.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.328615 25441 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:12:15.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.565369 25385 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:15.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.566084 25385 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: This master's current role is: LEADER
15:12:15.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.704675 25413 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:15.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.705173 25413 catalog_manager.cc:1562] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: loading cluster ID for follower catalog manager: success
15:12:15.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.712523 25413 catalog_manager.cc:1584] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: acquiring CA information for follower catalog manager: success
15:12:15.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.719163 25413 catalog_manager.cc:1612] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:12:17.580 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.24.132.188:45315
15:12:17.581 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 2
}
}
15:12:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:17.581744 25106 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 25297
15:12:17.611 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:17.611 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.24.132.190:36349
15:12:17.611 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 0
}
}
15:12:17.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:17.611770 25106 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 25134
15:12:17.635 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:17.636 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.24.132.189:40653
15:12:17.636 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 1
}
}
15:12:17.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:17.636449 25106 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 25213
15:12:17.663 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:17.663 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.24.132.188:45315
15:12:17.663 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 2
}
}
15:12:17.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:17.665438 25106 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:17.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:17.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/logs
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/info.pb
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.188:45315
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.188
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41693
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.188
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
15:12:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:12:17.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:17.970718 25448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:17.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:17.971246 25448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:17.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:17.971984 25448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:18.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.004331 25448 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
15:12:18.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.004637 25448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005059 25448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005296 25448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005494 25448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005686 25448 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005831 25448 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45445/jwks.json
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:18.047106 25448 master_runner.cc:386] Master server non-default flags:
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.188
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.188:45315
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/info.pb
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.188
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41693
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/logs
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:18.048249 25448 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:18.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:18.049167 25448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:18.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.059994 25455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:19.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.060727 25456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:19.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.363989 25458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:19.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.365658 25448 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.306s user 0.501s sys 0.801s
15:12:19.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.365921 25448 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.306s user 0.501s sys 0.801s
15:12:19.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.370180 25457 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1301 milliseconds
15:12:19.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.370242 25448 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:12:19.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.371592 25448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:19.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.374189 25448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:19.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.375533 25448 hybrid_clock.cc:648] HybridClock initialized: now 1765725139375502 us; error 41 us; skew 500 ppm
15:12:19.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.376398 25448 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:19.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:19 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725139, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:19.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.383208 25448 init.cc:377] Logged in from keytab as kudu/127.24.132.188@KRBTEST.COM (short username kudu)
15:12:19.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.388180 25448 webserver.cc:492] Webserver started at http://127.24.132.188:41693/ using document root <none> and password file <none>
15:12:19.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.389067 25448 fs_manager.cc:362] Metadata directory not provided
15:12:19.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.389293 25448 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:19.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.397918 25448 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.003s
15:12:19.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.402453 25467 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:19.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.403578 25448 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.403873 25448 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d4a2215e94e24ae4b512a89aa50d00fa"
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:14 on dist-test-slave-6vpz"
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.406015 25448 fs_report.cc:389] FS layout report
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/data
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:19.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:19.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:19.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.485404 25448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:19.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.495004 25448 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:19.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.495604 25448 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:19.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.567468 25448 rpc_server.cc:307] RPC server started. Bound to: 127.24.132.188:45315
15:12:19.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.567545 25518 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.132.188:45315 every 8 connection(s)
15:12:19.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.570355 25448 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/info.pb
15:12:19.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.579214 25106 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 25448
15:12:19.579 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:19.580 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.24.132.190:36349
15:12:19.580 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 0
}
}
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.581691 25519 sys_catalog.cc:263] Verifying existing consensus state
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.581756 25106 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/logs
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/info.pb
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.190:36349
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.190
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45571
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.190
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:12:19.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.589529 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Bootstrap starting.
15:12:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.628113 25519 log.cc:826] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Log is configured to *not* fsync() on all Append() calls
15:12:19.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.644322 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
15:12:19.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.644994 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Bootstrap complete.
15:12:19.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.665853 25519 raft_consensus.cc:359] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:19.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.668301 25519 raft_consensus.cc:740] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a2215e94e24ae4b512a89aa50d00fa, State: Initialized, Role: FOLLOWER
15:12:19.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.669142 25519 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:19.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.671970 25523 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:19.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.672787 25523 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: This master's current role is: FOLLOWER
15:12:19.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.673650 25519 sys_catalog.cc:565] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: configured and running, proceeding with master startup.
15:12:19.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.703749 25534 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:19.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.704247 25534 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: loading cluster ID for follower catalog manager: success
15:12:19.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.712159 25534 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: acquiring CA information for follower catalog manager: success
15:12:19.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.718102 25534 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:12:19.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.700312 25448 master_runner.cc:186] Error getting master registration for 127.24.132.190:36349: Network error: Client connection negotiation failed: client connection to 127.24.132.190:36349: connect: Connection refused (error 111),
15:12:19.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.725440 25448 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:12:19.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.914695 25521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:19.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.915274 25521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:19.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.916011 25521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.947438 25521 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
15:12:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.947636 25521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.947866 25521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.948041 25521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.948292 25521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.948482 25521 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.948612 25521 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45445/jwks.json
15:12:19.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.986536 25521 master_runner.cc:386] Master server non-default flags:
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.190
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.190:36349
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/info.pb
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.190
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45571
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/logs
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.987830 25521 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:19.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.988814 25521 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:20.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.999444 25542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:21.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:20.006296 25545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:21.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:20.003314 25543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:21.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:21.414235 25544 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1407 milliseconds
15:12:21.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.414363 25521 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:12:21.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.415743 25521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:21.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.419209 25521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:21.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.420646 25521 hybrid_clock.cc:648] HybridClock initialized: now 1765725141420595 us; error 84 us; skew 500 ppm
15:12:21.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.421492 25521 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:21.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:21 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725141, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:21.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.427660 25521 init.cc:377] Logged in from keytab as kudu/127.24.132.190@KRBTEST.COM (short username kudu)
15:12:21.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.433993 25521 webserver.cc:492] Webserver started at http://127.24.132.190:45571/ using document root <none> and password file <none>
15:12:21.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.435099 25521 fs_manager.cc:362] Metadata directory not provided
15:12:21.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.435405 25521 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:21.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.444226 25521 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.004s
15:12:21.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.450044 25555 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:21.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.451213 25521 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
15:12:21.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.451506 25521 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:21.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "53c07e78e2ea4944abad82dc6a1d4931"
15:12:21.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:10 on dist-test-slave-6vpz"
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.453486 25521 fs_report.cc:389] FS layout report
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/data
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:21.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.479501 25556 raft_consensus.cc:493] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
15:12:21.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.480019 25556 raft_consensus.cc:515] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:21.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.485047 25556 leader_election.cc:290] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53c07e78e2ea4944abad82dc6a1d4931 (127.24.132.190:36349), cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653)
15:12:21.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:21.488711 25471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.132.190:36349: connect: Connection refused (error 111)
15:12:21.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:21.493407 25471 leader_election.cc:336] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 53c07e78e2ea4944abad82dc6a1d4931 (127.24.132.190:36349): Network error: Client connection negotiation failed: client connection to 127.24.132.190:36349: connect: Connection refused (error 111)
15:12:21.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:21.494419 25469 leader_election.cc:336] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653): Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:21.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.494931 25469 leader_election.cc:304] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a2215e94e24ae4b512a89aa50d00fa; no voters: 53c07e78e2ea4944abad82dc6a1d4931, cfea2b69ea4d4c4a9a6837eb55774ae0
15:12:21.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.496363 25556 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
15:12:21.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.513373 25521 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:21.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.519672 25521 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:21.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.520233 25521 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:21.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.613556 25521 rpc_server.cc:307] RPC server started. Bound to: 127.24.132.190:36349
15:12:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.613631 25610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.132.190:36349 every 8 connection(s)
15:12:21.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.616478 25521 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/info.pb
15:12:21.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.625711 25106 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 25521
15:12:21.626 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:21.626 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.24.132.189:40653
15:12:21.626 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 1
}
}
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.628283 25106 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/logs
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/info.pb
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.189:40653
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.189
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34225
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.189
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.629547 25611 sys_catalog.cc:263] Verifying existing consensus state
15:12:21.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.636662 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Bootstrap starting.
15:12:21.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.672662 25611 log.cc:826] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Log is configured to *not* fsync() on all Append() calls
15:12:21.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.687310 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
15:12:21.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.688105 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Bootstrap complete.
15:12:21.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.709476 25611 raft_consensus.cc:359] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:21.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.711678 25611 raft_consensus.cc:740] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c07e78e2ea4944abad82dc6a1d4931, State: Initialized, Role: FOLLOWER
15:12:21.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.712601 25611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:21.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.715384 25615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:21.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.716375 25615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: FOLLOWER
15:12:21.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.717427 25611 sys_catalog.cc:565] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: configured and running, proceeding with master startup.
15:12:21.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.777666 25627 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:21.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.777963 25627 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: loading cluster ID for follower catalog manager: success
15:12:21.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.785459 25627 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: acquiring CA information for follower catalog manager: success
15:12:21.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:21 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725141, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.190@KRBTEST.COM for kudu/127.24.132.190@KRBTEST.COM
15:12:21.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.790601 25627 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:12:21.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.813221 25521 master_runner.cc:186] Error getting master registration for 127.24.132.189:40653: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111),
15:12:21.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.871994 25521 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:12:22.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.070969 25613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:22.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.071935 25613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:22.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.073133 25613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:22.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.134161 25613 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
15:12:22.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.134469 25613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:12:22.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.134922 25613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:22.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.135293 25613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:12:22.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.135634 25613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:12:22.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.136040 25613 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:12:22.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.136310 25613 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45445/jwks.json
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:22.202206 25613 master_runner.cc:386] Master server non-default flags:
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.189
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.189:40653
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/info.pb
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.189
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34225
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/logs
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:22.203781 25613 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:22.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:22.204998 25613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:22.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.218295 25636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:23.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.374578 25642 raft_consensus.cc:493] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
15:12:23.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.375514 25642 raft_consensus.cc:515] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:23.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.393575 25642 leader_election.cc:290] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653), d4a2215e94e24ae4b512a89aa50d00fa (127.24.132.188:45315)
15:12:23.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.394549 25561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:23.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.405519 25561 leader_election.cc:336] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653): Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:23.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.416551 25494 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53c07e78e2ea4944abad82dc6a1d4931" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" is_pre_election: true
15:12:23.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.417461 25494 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53c07e78e2ea4944abad82dc6a1d4931 in term 1.
15:12:23.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.421586 25562 leader_election.cc:304] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53c07e78e2ea4944abad82dc6a1d4931, d4a2215e94e24ae4b512a89aa50d00fa; no voters: cfea2b69ea4d4c4a9a6837eb55774ae0
15:12:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.423069 25642 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Leader pre-election won for term 2
15:12:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.423477 25642 raft_consensus.cc:493] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
15:12:23.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.423846 25642 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Advancing to term 2
15:12:23.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.435166 25642 raft_consensus.cc:515] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:23.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.441318 25561 leader_election.cc:336] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653): Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:23.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.443399 25494 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53c07e78e2ea4944abad82dc6a1d4931" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a2215e94e24ae4b512a89aa50d00fa"
15:12:23.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.443948 25494 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Advancing to term 2
15:12:23.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.454635 25642 leader_election.cc:290] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 election: Requested vote from peers cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653), d4a2215e94e24ae4b512a89aa50d00fa (127.24.132.188:45315)
15:12:23.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.456370 25494 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c07e78e2ea4944abad82dc6a1d4931 in term 2.
15:12:23.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.457604 25562 leader_election.cc:304] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53c07e78e2ea4944abad82dc6a1d4931, d4a2215e94e24ae4b512a89aa50d00fa; no voters: cfea2b69ea4d4c4a9a6837eb55774ae0
15:12:23.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.458644 25642 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 2 FOLLOWER]: Leader election won for term 2
15:12:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.464550 25642 raft_consensus.cc:697] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 2 LEADER]: Becoming Leader. State: Replica: 53c07e78e2ea4944abad82dc6a1d4931, State: Running, Role: LEADER
15:12:23.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.465499 25642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:23.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.480492 25647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c07e78e2ea4944abad82dc6a1d4931. Latest consensus state: current_term: 2 leader_uuid: "53c07e78e2ea4944abad82dc6a1d4931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:23.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.481451 25647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: LEADER
15:12:23.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.494287 25649 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
15:12:23.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.502856 25649 catalog_manager.cc:1494] Initializing Kudu cluster ID...
15:12:23.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.506880 25649 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:23.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.507140 25649 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
15:12:23.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.512351 25649 catalog_manager.cc:1514] Loading token signing keys...
15:12:23.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.520992 25649 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Loaded TSK: 0
15:12:23.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.522305 25649 catalog_manager.cc:1524] Initializing in-progress tserver states...
15:12:23.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.619637 25635 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ca0 after lost signal to thread 25613
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.711859 25635 kernel_stack_watchdog.cc:198] Thread 25613 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 394ms:
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.220979 25637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:23.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.721180 25613 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.498s user 0.002s sys 0.000s
15:12:23.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.721606 25613 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.498s user 0.002s sys 0.000s
15:12:23.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.723213 25639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:23.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.723402 25613 server_base.cc:1047] running on GCE node
15:12:23.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.724716 25613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:23.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.727129 25613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:23.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.728513 25613 hybrid_clock.cc:648] HybridClock initialized: now 1765725143728443 us; error 72 us; skew 500 ppm
15:12:23.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.729368 25613 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:23.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:23 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725143, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:23.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.736028 25613 init.cc:377] Logged in from keytab as kudu/127.24.132.189@KRBTEST.COM (short username kudu)
15:12:23.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.742038 25613 webserver.cc:492] Webserver started at http://127.24.132.189:34225/ using document root <none> and password file <none>
15:12:23.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.743299 25613 fs_manager.cc:362] Metadata directory not provided
15:12:23.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.743592 25613 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:23.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.752440 25613 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
15:12:23.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.757129 25656 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.758571 25613 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
15:12:23.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.758953 25613 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:23.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0"
15:12:23.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:12 on dist-test-slave-6vpz"
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.761613 25613 fs_report.cc:389] FS layout report
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/data
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:23.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.822003 25613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:23.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.829385 25613 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:23.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.829911 25613 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:23.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.923925 25561 consensus_peers.cc:597] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 -> Peer cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653): Couldn't send request to peer cfea2b69ea4d4c4a9a6837eb55774ae0. Status: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
15:12:23.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.952184 25613 rpc_server.cc:307] RPC server started. Bound to: 127.24.132.189:40653
15:12:23.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.952219 25707 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.132.189:40653 every 8 connection(s)
15:12:23.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.955983 25613 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/info.pb
15:12:23.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.961740 25106 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 25613
15:12:23.962 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:23.963 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.24.132.188:45315(127.24.132.188:45315)] connecting to peer
15:12:23.965 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.24.132.190:36349(127.24.132.190:36349)] connecting to peer
15:12:23.967 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.24.132.189:40653(127.24.132.189:40653)] connecting to peer
15:12:23.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.970239 25708 sys_catalog.cc:263] Verifying existing consensus state
15:12:23.971 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.24.132.190:36349(127.24.132.190:36349)] Successfully connected to peer
15:12:23.980 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.24.132.188:45315(127.24.132.188:45315)] Successfully connected to peer
15:12:23.981 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.24.132.189:40653(127.24.132.189:40653)] Successfully connected to peer
15:12:23.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.981603 25708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Bootstrap starting.
15:12:23.985 [DEBUG - kudu-nio-1] (SecurityContext.java:222) Refreshing Kerberos credentials...
15:12:23.986 [DEBUG - kudu-nio-1] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc
15:12:24.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:24 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:24.005 [DEBUG - kudu-nio-1] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
15:12:24.005 [WARN - kudu-nio-1] (SecurityContext.java:231) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
15:12:24.005 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.24.132.190:36349(127.24.132.190:36349)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.006 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.24.132.190:36349: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.006 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.24.132.190:36349(127.24.132.190:36349)] handling channelInactive
15:12:24.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:24.008308 25714 negotiation.cc:337] Failed RPC negotiation. Trace:
15:12:24.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.972322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60060 (local address 127.24.132.190:36349)
15:12:24.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983300 (+ 10978us) server_negotiation.cc:207] Beginning negotiation
15:12:24.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983338 (+ 38us) server_negotiation.cc:400] Waiting for connection header
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983390 (+ 52us) server_negotiation.cc:408] Connection header received
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983532 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983559 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983775 (+ 216us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.001560 (+ 17785us) server_negotiation.cc:227] Negotiated authn=SASL
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.006915 (+ 5355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60060: BlockingRecv error: recv got EOF from 127.0.0.1:60060 (error 108)
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":10330,"thread_start_us":10054,"threads_started":1}
15:12:24.011 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.24.132.188:45315(127.24.132.188:45315)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.012 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.24.132.188:45315: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.012 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.24.132.188:45315(127.24.132.188:45315)] handling channelInactive
15:12:24.013 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.24.132.189:40653(127.24.132.189:40653)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.013 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.24.132.189:40653: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:24.015595 25713 negotiation.cc:337] Failed RPC negotiation. Trace:
15:12:24.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.968097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.24.132.188:45315)
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005607 (+ 37510us) server_negotiation.cc:207] Beginning negotiation
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005639 (+ 32us) server_negotiation.cc:400] Waiting for connection header
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005688 (+ 49us) server_negotiation.cc:408] Connection header received
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005824 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005849 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.006139 (+ 290us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.013562 (+ 7423us) server_negotiation.cc:227] Negotiated authn=SASL
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.014158 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48212: BlockingRecv error: recv got EOF from 127.0.0.1:48212 (error 108)
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":36915,"thread_start_us":38878,"threads_started":1}
15:12:24.015 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1523) Notify RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.188:45315, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.190:36349, [5ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.189:40653, [44ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.190:36349: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [49ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.188:45315: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [51ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.189:40653: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) after lookup exception
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.24.132.188:45315,127.24.132.190:36349,127.24.132.189:40653). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268) [main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) [main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) [main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) [main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) [main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) [main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) [main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) [main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289) [netty-handler-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.24.132.188:45315,127.24.132.190:36349,127.24.132.189:40653). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
15:12:24.019 [DEBUG - kudu-nio-0] (KuduRpc.java:260) Handling a callback on RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=56), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.188:45315, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.190:36349, [5ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.189:40653, [44ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.190:36349: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [49ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.188:45315: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [51ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.189:40653: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) with no deferred attached!
"
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) ~[hamcrest-core-1.3.jar:1.3]
at org.junit.Assert.assertThat(Assert.java:964) ~[junit-4.13.2.jar:4.13.2]
at org.junit.Assert.assertThat(Assert.java:930) ~[junit-4.13.2.jar:4.13.2]
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:370) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) ~[junit-4.13.2.jar:4.13.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
java.lang.AssertionError:
Expected: a string containing "found that the new Kerberos principal test-user@KRBTEST.COM did not match the original principal test-admin@KRBTEST.COM"
but: was "15:12:14.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.559465 25297 master_runner.cc:186] Error getting master registration for 127.24.132.190:36349: OK, instance_id { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" instance_seqno: 1765725130235270 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
15:12:14.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.589640 25297 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:12:14.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.591519 25287 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 7
15:12:14.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:14 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725131, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.189@KRBTEST.COM for kudu/127.24.132.188@KRBTEST.COM
15:12:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.622637 25287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Bootstrap starting.
15:12:14.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.630120 25287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Neither blocks nor log segments found. Creating new log.
15:12:14.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.632421 25287 log.cc:826] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Log is configured to *not* fsync() on all Append() calls
15:12:14.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.637287 25287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: No bootstrap required, opened a new log
15:12:14.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.663478 25287 raft_consensus.cc:359] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.664309 25287 raft_consensus.cc:385] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
15:12:14.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.664616 25287 raft_consensus.cc:740] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfea2b69ea4d4c4a9a6837eb55774ae0, State: Initialized, Role: FOLLOWER
15:12:14.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.665508 25287 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.668221 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.668767 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: This master's current role is: FOLLOWER
15:12:14.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.670257 25287 sys_catalog.cc:565] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: configured and running, proceeding with master startup.
15:12:14.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.698362 25413 catalog_manager.cc:1568] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
15:12:14.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.698673 25413 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
15:12:14.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.692229 25213 master_runner.cc:186] Error getting master registration for 127.24.132.190:36349: OK, instance_id { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" instance_seqno: 1765725130235270 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
15:12:14.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.714720 25213 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:12:14.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.799407 25385 raft_consensus.cc:493] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
15:12:14.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.799933 25385 raft_consensus.cc:515] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.802333 25385 leader_election.cc:290] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c07e78e2ea4944abad82dc6a1d4931 (127.24.132.190:36349), cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653)
15:12:14.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.804334 25262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" is_pre_election: true
15:12:14.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.805236 25262 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a2215e94e24ae4b512a89aa50d00fa in term 0.
15:12:14.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.806581 25324 leader_election.cc:304] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfea2b69ea4d4c4a9a6837eb55774ae0, d4a2215e94e24ae4b512a89aa50d00fa; no voters:
15:12:14.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.807458 25385 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Leader pre-election won for term 1
15:12:14.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.807737 25385 raft_consensus.cc:493] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
15:12:14.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.808019 25385 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 0 FOLLOWER]: Advancing to term 1
15:12:14.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.815591 25385 raft_consensus.cc:515] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.818835 25385 leader_election.cc:290] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 1 election: Requested vote from peers 53c07e78e2ea4944abad82dc6a1d4931 (127.24.132.190:36349), cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653)
15:12:14.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.819681 25262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0"
15:12:14.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.820192 25262 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 0 FOLLOWER]: Advancing to term 1
15:12:14.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.829993 25262 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a2215e94e24ae4b512a89aa50d00fa in term 1.
15:12:14.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.831763 25324 leader_election.cc:304] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfea2b69ea4d4c4a9a6837eb55774ae0, d4a2215e94e24ae4b512a89aa50d00fa; no voters:
15:12:14.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.832597 25385 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Leader election won for term 1
15:12:14.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.855141 25385 raft_consensus.cc:697] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 LEADER]: Becoming Leader. State: Replica: d4a2215e94e24ae4b512a89aa50d00fa, State: Running, Role: LEADER
15:12:14.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.856331 25385 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:14.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:14.861999 25241 tablet.cc:2378] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Can't schedule compaction. Clean time has not been advanced past its initial value.
15:12:14.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.878578 25415 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a2215e94e24ae4b512a89aa50d00fa. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.879137 25415 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: This master's current role is: LEADER
15:12:14.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.893849 25417 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
15:12:14.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.908516 25417 catalog_manager.cc:1494] Initializing Kudu cluster ID...
15:12:14.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.933740 25262 raft_consensus.cc:1275] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [term 1 FOLLOWER]: Refusing update from remote peer d4a2215e94e24ae4b512a89aa50d00fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
15:12:14.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.935312 25415 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
15:12:14.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.947091 25204 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } attempt: 7
15:12:14.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.968827 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a2215e94e24ae4b512a89aa50d00fa. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.969736 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: This master's current role is: FOLLOWER
15:12:14.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.977600 25418 mvcc.cc:204] Tried to move back new op lower bound from 7232410152671993856 to 7232410152381349888. Current Snapshot: MvccSnapshot[applied={T|T < 7232410152671993856}]
15:12:14.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.989781 25385 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.991524 25385 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: This master's current role is: LEADER
15:12:14.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.996289 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:14.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:14.996965 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0 [sys.catalog]: This master's current role is: FOLLOWER
15:12:15.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.003165 25417 catalog_manager.cc:1357] Generated new cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:15.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.003427 25417 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
15:12:15.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:15 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725130, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.190@KRBTEST.COM for kudu/127.24.132.188@KRBTEST.COM
15:12:15.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.045742 25204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Bootstrap starting.
15:12:15.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.054105 25204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Neither blocks nor log segments found. Creating new log.
15:12:15.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.056912 25204 log.cc:826] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Log is configured to *not* fsync() on all Append() calls
15:12:15.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.071456 25417 catalog_manager.cc:1380] Generated new certificate authority record
15:12:15.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.075678 25417 catalog_manager.cc:1514] Loading token signing keys...
15:12:15.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.083077 25204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: No bootstrap required, opened a new log
15:12:15.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.120469 25417 catalog_manager.cc:6027] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Generated new TSK 0
15:12:15.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.121562 25417 catalog_manager.cc:1524] Initializing in-progress tserver states...
15:12:15.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.140190 25204 raft_consensus.cc:359] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:15.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.141217 25204 raft_consensus.cc:385] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
15:12:15.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.141691 25204 raft_consensus.cc:740] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c07e78e2ea4944abad82dc6a1d4931, State: Initialized, Role: FOLLOWER
15:12:15.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.142678 25204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:15.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.146044 25425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:15.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.147066 25425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: FOLLOWER
15:12:15.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.147935 25204 sys_catalog.cc:565] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: configured and running, proceeding with master startup.
15:12:15.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.148138 25178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c07e78e2ea4944abad82dc6a1d4931"
15:12:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.150911 25425 raft_consensus.cc:493] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
15:12:15.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.148860 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c07e78e2ea4944abad82dc6a1d4931" is_pre_election: true
15:12:15.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.151968 25425 raft_consensus.cc:515] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:15.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.153277 25177 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 0 FOLLOWER]: Advancing to term 1
15:12:15.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.154480 25425 leader_election.cc:290] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653), d4a2215e94e24ae4b512a89aa50d00fa (127.24.132.188:45315)
15:12:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.157774 25262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53c07e78e2ea4944abad82dc6a1d4931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" is_pre_election: true
15:12:15.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.158785 25349 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53c07e78e2ea4944abad82dc6a1d4931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" is_pre_election: true
15:12:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.160447 25155 leader_election.cc:304] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53c07e78e2ea4944abad82dc6a1d4931; no voters: cfea2b69ea4d4c4a9a6837eb55774ae0, d4a2215e94e24ae4b512a89aa50d00fa
15:12:15.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.166872 25177 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Refusing update from remote peer d4a2215e94e24ae4b512a89aa50d00fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
15:12:15.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.167514 25425 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
15:12:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.181799 25385 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
15:12:15.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:15.195878 25158 tablet.cc:2378] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Can't schedule compaction. Clean time has not been advanced past its initial value.
15:12:15.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.250007 25431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:15.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.250753 25431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: FOLLOWER
15:12:15.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.252115 25425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a2215e94e24ae4b512a89aa50d00fa. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:15.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.252805 25425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: FOLLOWER
15:12:15.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.315683 25441 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:15.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.316200 25441 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: loading cluster ID for follower catalog manager: success
15:12:15.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.322973 25441 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: acquiring CA information for follower catalog manager: success
15:12:15.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.328615 25441 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:12:15.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.565369 25385 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:15.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.566084 25385 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: This master's current role is: LEADER
15:12:15.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.704675 25413 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:15.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.705173 25413 catalog_manager.cc:1562] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: loading cluster ID for follower catalog manager: success
15:12:15.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.712523 25413 catalog_manager.cc:1584] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: acquiring CA information for follower catalog manager: success
15:12:15.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:15.719163 25413 catalog_manager.cc:1612] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:12:17.580 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.24.132.188:45315
15:12:17.581 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 2
}
}
15:12:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:17.581744 25106 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 25297
15:12:17.611 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:17.611 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.24.132.190:36349
15:12:17.611 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 0
}
}
15:12:17.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:17.611770 25106 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 25134
15:12:17.635 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:17.636 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.24.132.189:40653
15:12:17.636 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 1
}
}
15:12:17.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:17.636449 25106 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu with pid 25213
15:12:17.663 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:17.663 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.24.132.188:45315
15:12:17.663 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 2
}
}
15:12:17.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:17.665438 25106 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:17.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:17.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/logs
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/info.pb
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.188:45315
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.188
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41693
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.188
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
15:12:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:12:17.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:17.970718 25448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:17.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:17.971246 25448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:17.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:17.971984 25448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:18.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.004331 25448 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
15:12:18.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.004637 25448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005059 25448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005296 25448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005494 25448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005686 25448 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:12:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.005831 25448 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45445/jwks.json
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:18.047106 25448 master_runner.cc:386] Master server non-default flags:
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.188
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.188:45315
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/info.pb
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.188
15:12:18.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41693
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/logs
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:18.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:18.048249 25448 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:18.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:18.049167 25448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:18.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.059994 25455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:19.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:18.060727 25456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:19.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.363989 25458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:19.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.365658 25448 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.306s user 0.501s sys 0.801s
15:12:19.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.365921 25448 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.306s user 0.501s sys 0.801s
15:12:19.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.370180 25457 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1301 milliseconds
15:12:19.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.370242 25448 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:12:19.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.371592 25448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:19.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.374189 25448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:19.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.375533 25448 hybrid_clock.cc:648] HybridClock initialized: now 1765725139375502 us; error 41 us; skew 500 ppm
15:12:19.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.376398 25448 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:19.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:19 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725139, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:19.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.383208 25448 init.cc:377] Logged in from keytab as kudu/127.24.132.188@KRBTEST.COM (short username kudu)
15:12:19.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.388180 25448 webserver.cc:492] Webserver started at http://127.24.132.188:41693/ using document root <none> and password file <none>
15:12:19.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.389067 25448 fs_manager.cc:362] Metadata directory not provided
15:12:19.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.389293 25448 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:19.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.397918 25448 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.003s
15:12:19.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.402453 25467 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:19.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.403578 25448 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.403873 25448 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d4a2215e94e24ae4b512a89aa50d00fa"
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:14 on dist-test-slave-6vpz"
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.406015 25448 fs_report.cc:389] FS layout report
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/wal
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/data
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:19.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:19.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:19.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:19.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.485404 25448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:19.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.495004 25448 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:19.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.495604 25448 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:19.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.567468 25448 rpc_server.cc:307] RPC server started. Bound to: 127.24.132.188:45315
15:12:19.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.567545 25518 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.132.188:45315 every 8 connection(s)
15:12:19.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.570355 25448 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-2/data/info.pb
15:12:19.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.579214 25106 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 25448
15:12:19.579 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:19.580 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.24.132.190:36349
15:12:19.580 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 0
}
}
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.581691 25519 sys_catalog.cc:263] Verifying existing consensus state
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.581756 25106 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/logs
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/info.pb
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.190:36349
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.190
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45571
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.190
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
15:12:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:12:19.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.589529 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Bootstrap starting.
15:12:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.628113 25519 log.cc:826] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Log is configured to *not* fsync() on all Append() calls
15:12:19.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.644322 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
15:12:19.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.644994 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: Bootstrap complete.
15:12:19.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.665853 25519 raft_consensus.cc:359] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:19.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.668301 25519 raft_consensus.cc:740] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a2215e94e24ae4b512a89aa50d00fa, State: Initialized, Role: FOLLOWER
15:12:19.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.669142 25519 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:19.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.671970 25523 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:19.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.672787 25523 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: This master's current role is: FOLLOWER
15:12:19.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.673650 25519 sys_catalog.cc:565] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [sys.catalog]: configured and running, proceeding with master startup.
15:12:19.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.703749 25534 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:19.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.704247 25534 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: loading cluster ID for follower catalog manager: success
15:12:19.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.712159 25534 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: acquiring CA information for follower catalog manager: success
15:12:19.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.718102 25534 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:12:19.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.700312 25448 master_runner.cc:186] Error getting master registration for 127.24.132.190:36349: Network error: Client connection negotiation failed: client connection to 127.24.132.190:36349: connect: Connection refused (error 111),
15:12:19.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.725440 25448 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:12:19.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.914695 25521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:19.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.915274 25521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:19.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.916011 25521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.947438 25521 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
15:12:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.947636 25521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.947866 25521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.948041 25521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.948292 25521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.948482 25521 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:12:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.948612 25521 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45445/jwks.json
15:12:19.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.986536 25521 master_runner.cc:386] Master server non-default flags:
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.190
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.190:36349
15:12:19.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/info.pb
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.190
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45571
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/logs
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.987830 25521 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:19.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:19.988814 25521 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:20.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:19.999444 25542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:21.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:20.006296 25545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:21.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:20.003314 25543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:21.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:21.414235 25544 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1407 milliseconds
15:12:21.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.414363 25521 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
15:12:21.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.415743 25521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:21.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.419209 25521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:21.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.420646 25521 hybrid_clock.cc:648] HybridClock initialized: now 1765725141420595 us; error 84 us; skew 500 ppm
15:12:21.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.421492 25521 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:21.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:21 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725141, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:21.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.427660 25521 init.cc:377] Logged in from keytab as kudu/127.24.132.190@KRBTEST.COM (short username kudu)
15:12:21.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.433993 25521 webserver.cc:492] Webserver started at http://127.24.132.190:45571/ using document root <none> and password file <none>
15:12:21.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.435099 25521 fs_manager.cc:362] Metadata directory not provided
15:12:21.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.435405 25521 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:21.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.444226 25521 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.004s
15:12:21.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.450044 25555 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:21.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.451213 25521 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
15:12:21.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.451506 25521 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:21.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "53c07e78e2ea4944abad82dc6a1d4931"
15:12:21.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:10 on dist-test-slave-6vpz"
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.453486 25521 fs_report.cc:389] FS layout report
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/wal
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/data
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:21.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:21.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.479501 25556 raft_consensus.cc:493] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
15:12:21.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.480019 25556 raft_consensus.cc:515] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:21.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.485047 25556 leader_election.cc:290] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53c07e78e2ea4944abad82dc6a1d4931 (127.24.132.190:36349), cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653)
15:12:21.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:21.488711 25471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.132.190:36349: connect: Connection refused (error 111)
15:12:21.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:21.493407 25471 leader_election.cc:336] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 53c07e78e2ea4944abad82dc6a1d4931 (127.24.132.190:36349): Network error: Client connection negotiation failed: client connection to 127.24.132.190:36349: connect: Connection refused (error 111)
15:12:21.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:21.494419 25469 leader_election.cc:336] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653): Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:21.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.494931 25469 leader_election.cc:304] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a2215e94e24ae4b512a89aa50d00fa; no voters: 53c07e78e2ea4944abad82dc6a1d4931, cfea2b69ea4d4c4a9a6837eb55774ae0
15:12:21.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.496363 25556 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
15:12:21.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.513373 25521 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:21.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.519672 25521 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:21.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.520233 25521 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:21.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.613556 25521 rpc_server.cc:307] RPC server started. Bound to: 127.24.132.190:36349
15:12:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.613631 25610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.132.190:36349 every 8 connection(s)
15:12:21.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.616478 25521 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-0/data/info.pb
15:12:21.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.625711 25106 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 25521
15:12:21.626 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:21.626 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.24.132.189:40653
15:12:21.626 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 1
}
}
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.628283 25106 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:12:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/logs
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/info.pb
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.189:40653
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.189
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34225
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:21.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.189
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:12:21.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.629547 25611 sys_catalog.cc:263] Verifying existing consensus state
15:12:21.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.636662 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Bootstrap starting.
15:12:21.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.672662 25611 log.cc:826] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Log is configured to *not* fsync() on all Append() calls
15:12:21.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.687310 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
15:12:21.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.688105 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Bootstrap complete.
15:12:21.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.709476 25611 raft_consensus.cc:359] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:21.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.711678 25611 raft_consensus.cc:740] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c07e78e2ea4944abad82dc6a1d4931, State: Initialized, Role: FOLLOWER
15:12:21.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.712601 25611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:21.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.715384 25615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:21.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.716375 25615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: FOLLOWER
15:12:21.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.717427 25611 sys_catalog.cc:565] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: configured and running, proceeding with master startup.
15:12:21.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.777666 25627 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:21.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.777963 25627 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: loading cluster ID for follower catalog manager: success
15:12:21.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.785459 25627 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: acquiring CA information for follower catalog manager: success
15:12:21.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:21 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725141, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.190@KRBTEST.COM for kudu/127.24.132.190@KRBTEST.COM
15:12:21.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.790601 25627 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
15:12:21.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.813221 25521 master_runner.cc:186] Error getting master registration for 127.24.132.189:40653: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111),
15:12:21.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:21.871994 25521 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
15:12:22.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.070969 25613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:12:22.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.071935 25613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:12:22.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.073133 25613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:12:22.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.134161 25613 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
15:12:22.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.134469 25613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:12:22.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.134922 25613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:12:22.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.135293 25613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:12:22.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.135634 25613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:12:22.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.136040 25613 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:12:22.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.136310 25613 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45445/jwks.json
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:22.202206 25613 master_runner.cc:386] Master server non-default flags:
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.24.132.148:43589
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:22.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.24.132.190:36349,127.24.132.189:40653,127.24.132.188:45315
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/kudu.keytab
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.24.132.189
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.24.132.189:40653
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45445/jwks.json
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/info.pb
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.24.132.189
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34225
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/logs
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:12:22.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision dc741732bb6a40213e11a13869fad06a49a209a3
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 14 Dec 2025 14:43:21 UTC on 5fd53c4cbb9d
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9513
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:12:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:22.203781 25613 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:12:22.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:22.204998 25613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:12:22.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.218295 25636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:23.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.374578 25642 raft_consensus.cc:493] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
15:12:23.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.375514 25642 raft_consensus.cc:515] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:23.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.393575 25642 leader_election.cc:290] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653), d4a2215e94e24ae4b512a89aa50d00fa (127.24.132.188:45315)
15:12:23.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.394549 25561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:23.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.405519 25561 leader_election.cc:336] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653): Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:23.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.416551 25494 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53c07e78e2ea4944abad82dc6a1d4931" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" is_pre_election: true
15:12:23.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.417461 25494 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53c07e78e2ea4944abad82dc6a1d4931 in term 1.
15:12:23.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.421586 25562 leader_election.cc:304] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53c07e78e2ea4944abad82dc6a1d4931, d4a2215e94e24ae4b512a89aa50d00fa; no voters: cfea2b69ea4d4c4a9a6837eb55774ae0
15:12:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.423069 25642 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Leader pre-election won for term 2
15:12:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.423477 25642 raft_consensus.cc:493] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
15:12:23.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.423846 25642 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 1 FOLLOWER]: Advancing to term 2
15:12:23.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.435166 25642 raft_consensus.cc:515] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:23.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.441318 25561 leader_election.cc:336] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653): Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111)
15:12:23.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.443399 25494 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53c07e78e2ea4944abad82dc6a1d4931" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a2215e94e24ae4b512a89aa50d00fa"
15:12:23.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.443948 25494 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 1 FOLLOWER]: Advancing to term 2
15:12:23.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.454635 25642 leader_election.cc:290] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 election: Requested vote from peers cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653), d4a2215e94e24ae4b512a89aa50d00fa (127.24.132.188:45315)
15:12:23.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.456370 25494 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d4a2215e94e24ae4b512a89aa50d00fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c07e78e2ea4944abad82dc6a1d4931 in term 2.
15:12:23.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.457604 25562 leader_election.cc:304] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53c07e78e2ea4944abad82dc6a1d4931, d4a2215e94e24ae4b512a89aa50d00fa; no voters: cfea2b69ea4d4c4a9a6837eb55774ae0
15:12:23.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.458644 25642 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 2 FOLLOWER]: Leader election won for term 2
15:12:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.464550 25642 raft_consensus.cc:697] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [term 2 LEADER]: Becoming Leader. State: Replica: 53c07e78e2ea4944abad82dc6a1d4931, State: Running, Role: LEADER
15:12:23.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.465499 25642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } }
15:12:23.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.480492 25647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c07e78e2ea4944abad82dc6a1d4931. Latest consensus state: current_term: 2 leader_uuid: "53c07e78e2ea4944abad82dc6a1d4931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c07e78e2ea4944abad82dc6a1d4931" member_type: VOTER last_known_addr { host: "127.24.132.190" port: 36349 } } peers { permanent_uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0" member_type: VOTER last_known_addr { host: "127.24.132.189" port: 40653 } } peers { permanent_uuid: "d4a2215e94e24ae4b512a89aa50d00fa" member_type: VOTER last_known_addr { host: "127.24.132.188" port: 45315 } } }
15:12:23.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.481451 25647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 [sys.catalog]: This master's current role is: LEADER
15:12:23.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.494287 25649 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
15:12:23.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.502856 25649 catalog_manager.cc:1494] Initializing Kudu cluster ID...
15:12:23.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.506880 25649 catalog_manager.cc:1269] Loaded cluster ID: 8fd741790430444cbc4161d51bf7f08d
15:12:23.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.507140 25649 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
15:12:23.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.512351 25649 catalog_manager.cc:1514] Loading token signing keys...
15:12:23.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.520992 25649 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931: Loaded TSK: 0
15:12:23.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.522305 25649 catalog_manager.cc:1524] Initializing in-progress tserver states...
15:12:23.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.619637 25635 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ca0 after lost signal to thread 25613
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.711859 25635 kernel_stack_watchdog.cc:198] Thread 25613 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 394ms:
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
15:12:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:22.220979 25637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:23.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.721180 25613 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.498s user 0.002s sys 0.000s
15:12:23.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.721606 25613 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.498s user 0.002s sys 0.000s
15:12:23.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.723213 25639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:12:23.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.723402 25613 server_base.cc:1047] running on GCE node
15:12:23.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.724716 25613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:12:23.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.727129 25613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:12:23.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.728513 25613 hybrid_clock.cc:648] HybridClock initialized: now 1765725143728443 us; error 72 us; skew 500 ppm
15:12:23.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.729368 25613 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:12:23.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:23 dist-test-slave-6vpz krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725143, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.132.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:23.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.736028 25613 init.cc:377] Logged in from keytab as kudu/127.24.132.189@KRBTEST.COM (short username kudu)
15:12:23.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.742038 25613 webserver.cc:492] Webserver started at http://127.24.132.189:34225/ using document root <none> and password file <none>
15:12:23.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.743299 25613 fs_manager.cc:362] Metadata directory not provided
15:12:23.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.743592 25613 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:12:23.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.752440 25613 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
15:12:23.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.757129 25656 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:12:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.758571 25613 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
15:12:23.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.758953 25613 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data,/tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:23.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cfea2b69ea4d4c4a9a6837eb55774ae0"
15:12:23.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-14 15:12:12 on dist-test-slave-6vpz"
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.761613 25613 fs_report.cc:389] FS layout report
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/wal
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/data
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:12:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:12:23.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.822003 25613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:12:23.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.829385 25613 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:12:23.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.829911 25613 kserver.cc:163] Server-wide thread pool size limit: 3276
15:12:23.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:23.923925 25561 consensus_peers.cc:597] T 00000000000000000000000000000000 P 53c07e78e2ea4944abad82dc6a1d4931 -> Peer cfea2b69ea4d4c4a9a6837eb55774ae0 (127.24.132.189:40653): Couldn't send request to peer cfea2b69ea4d4c4a9a6837eb55774ae0. Status: Network error: Client connection negotiation failed: client connection to 127.24.132.189:40653: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
15:12:23.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.952184 25613 rpc_server.cc:307] RPC server started. Bound to: 127.24.132.189:40653
15:12:23.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.952219 25707 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.132.189:40653 every 8 connection(s)
15:12:23.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.955983 25613 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/master-1/data/info.pb
15:12:23.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.961740 25106 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskJesFA1/build/tsan/bin/kudu as pid 25613
15:12:23.962 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
15:12:23.963 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.24.132.188:45315(127.24.132.188:45315)] connecting to peer
15:12:23.965 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.24.132.190:36349(127.24.132.190:36349)] connecting to peer
15:12:23.967 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.24.132.189:40653(127.24.132.189:40653)] connecting to peer
15:12:23.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.970239 25708 sys_catalog.cc:263] Verifying existing consensus state
15:12:23.971 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.24.132.190:36349(127.24.132.190:36349)] Successfully connected to peer
15:12:23.980 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.24.132.188:45315(127.24.132.188:45315)] Successfully connected to peer
15:12:23.981 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.24.132.189:40653(127.24.132.189:40653)] Successfully connected to peer
15:12:23.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251214 15:12:23.981603 25708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea2b69ea4d4c4a9a6837eb55774ae0: Bootstrap starting.
15:12:23.985 [DEBUG - kudu-nio-1] (SecurityContext.java:222) Refreshing Kerberos credentials...
15:12:23.986 [DEBUG - kudu-nio-1] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskJesFA1/test-tmp/mini-kudu-cluster7029738690141119687/krb5kdc/krb5cc
15:12:24.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 14 15:12:24 dist-test-slave-6vpz krb5kdc[25117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765725134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:12:24.005 [DEBUG - kudu-nio-1] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
15:12:24.005 [WARN - kudu-nio-1] (SecurityContext.java:231) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
15:12:24.005 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.24.132.190:36349(127.24.132.190:36349)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.006 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.24.132.190:36349: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.006 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.24.132.190:36349(127.24.132.190:36349)] handling channelInactive
15:12:24.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:24.008308 25714 negotiation.cc:337] Failed RPC negotiation. Trace:
15:12:24.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.972322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60060 (local address 127.24.132.190:36349)
15:12:24.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983300 (+ 10978us) server_negotiation.cc:207] Beginning negotiation
15:12:24.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983338 (+ 38us) server_negotiation.cc:400] Waiting for connection header
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983390 (+ 52us) server_negotiation.cc:408] Connection header received
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983532 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983559 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.983775 (+ 216us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.001560 (+ 17785us) server_negotiation.cc:227] Negotiated authn=SASL
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.006915 (+ 5355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60060: BlockingRecv error: recv got EOF from 127.0.0.1:60060 (error 108)
15:12:24.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":10330,"thread_start_us":10054,"threads_started":1}
15:12:24.011 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.24.132.188:45315(127.24.132.188:45315)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.012 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.24.132.188:45315: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.012 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.24.132.188:45315(127.24.132.188:45315)] handling channelInactive
15:12:24.013 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.24.132.189:40653(127.24.132.189:40653)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.013 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.24.132.189:40653: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
15:12:24.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251214 15:12:24.015595 25713 negotiation.cc:337] Failed RPC negotiation. Trace:
15:12:24.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:23.968097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.24.132.188:45315)
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005607 (+ 37510us) server_negotiation.cc:207] Beginning negotiation
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005639 (+ 32us) server_negotiation.cc:400] Waiting for connection header
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005688 (+ 49us) server_negotiation.cc:408] Connection header received
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005824 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.005849 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.006139 (+ 290us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.013562 (+ 7423us) server_negotiation.cc:227] Negotiated authn=SASL
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1214 15:12:24.014158 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48212: BlockingRecv error: recv got EOF from 127.0.0.1:48212 (error 108)
15:12:24.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":36915,"thread_start_us":38878,"threads_started":1}
15:12:24.015 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1523) Notify RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.188:45315, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.190:36349, [5ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.189:40653, [44ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.190:36349: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [49ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.188:45315: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [51ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.189:40653: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) after lookup exception
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.24.132.188:45315,127.24.132.190:36349,127.24.132.189:40653). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268) [main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) [main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) [main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) [main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) [main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) [main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) [main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) [main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289) [netty-handler-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.115.Final.jar:4.1.115.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.24.132.188:45315,127.24.132.190:36349,127.24.132.189:40653). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
15:12:24.019 [DEBUG - kudu-nio-0] (KuduRpc.java:260) Handling a callback on RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=56), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.188:45315, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.190:36349, [5ms] Sub RPC ConnectToMaster: sending RPC to server master-127.24.132.189:40653, [44ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.190:36349: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [49ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.188:45315: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [51ms] Sub RPC ConnectToMaster: received response from server master-127.24.132.189:40653: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) with no deferred attached!
"
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:964)
at org.junit.Assert.assertThat(Assert.java:930)
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:370)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)