[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:13:20 dist-test-slave-l6qh krb5kdc[3156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:13:20 dist-test-slave-l6qh krb5kdc[3156](info): set up 2 sockets
Sep 25 09:13:20 dist-test-slave-l6qh krb5kdc[3156](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 09:13:21.591075 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:13:21 dist-test-slave-l6qh krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:13:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:13:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:13:21.757707 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:44599
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:35583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:44599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:22.238262 3169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:22.238586 3169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:22.238740 3169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:22.246515 3169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:13:22.246640 3169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:22.246739 3169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:22.246837 3169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:13:22.246908 3169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:13:22.256301 3169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:44599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:44599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:22.257614 3169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:22.258868 3169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:22.272357 3178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:22.272378 3175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:22.272490 3176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:22.273185 3169 server_base.cc:1034] running on GCE node
I20260925 09:13:22.274173 3169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:22.275609 3169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:22.276826 3169 hybrid_clock.cc:648] HybridClock initialized: now 1790327602276779 us; error 38 us; skew 500 ppm
I20260925 09:13:22.277197 3169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:22 dist-test-slave-l6qh krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:22.280901 3169 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:13:22.282562 3169 webserver.cc:460] Webserver started at http://127.3.19.126:44297/ using document root <none> and password file <none>
I20260925 09:13:22.283221 3169 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:22.283375 3169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:22.283735 3169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:22.286968 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "dd67e25423894730925d371d1da9433f"
format_stamp: "Formatted at 2026-09-25 09:13:22 on dist-test-slave-l6qh"
server_key: "9f2db5bbb0aa12544316cfc215298ac3"
server_key_iv: "0e2888bcad9f7b80839b6fcf510bc5e8"
server_key_version: "encryptionkey@0"
I20260925 09:13:22.287768 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "dd67e25423894730925d371d1da9433f"
format_stamp: "Formatted at 2026-09-25 09:13:22 on dist-test-slave-l6qh"
server_key: "9f2db5bbb0aa12544316cfc215298ac3"
server_key_iv: "0e2888bcad9f7b80839b6fcf510bc5e8"
server_key_version: "encryptionkey@0"
I20260925 09:13:22.293104 3169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 09:13:22.296567 3185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:22.297894 3169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:13:22.298064 3169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "dd67e25423894730925d371d1da9433f"
format_stamp: "Formatted at 2026-09-25 09:13:22 on dist-test-slave-l6qh"
server_key: "9f2db5bbb0aa12544316cfc215298ac3"
server_key_iv: "0e2888bcad9f7b80839b6fcf510bc5e8"
server_key_version: "encryptionkey@0"
I20260925 09:13:22.298249 3169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:22.317451 3169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:22.321379 3169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:22.321653 3169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:22.358204 3169 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:44599
I20260925 09:13:22.358363 3236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:44599 every 8 connection(s)
I20260925 09:13:22.359854 3169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:13:22.364573 3237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:22.365801 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 3169
I20260925 09:13:22.366204 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:13:22.368404 3149 external_mini_cluster.cc:1428] Setting key b5079f919a80387e693ce5e83f03a0e9
I20260925 09:13:22.379606 3237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f: Bootstrap starting.
I20260925 09:13:22.383458 3237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:22.384826 3237 log.cc:824] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:22.388399 3237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f: No bootstrap required, opened a new log
I20260925 09:13:22.397956 3237 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd67e25423894730925d371d1da9433f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44599 } }
I20260925 09:13:22.398311 3237 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:22.398423 3237 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd67e25423894730925d371d1da9433f, State: Initialized, Role: FOLLOWER
I20260925 09:13:22.399669 3237 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd67e25423894730925d371d1da9433f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44599 } }
I20260925 09:13:22.399936 3237 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:13:22.400072 3237 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:13:22.400235 3237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:22.404134 3237 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd67e25423894730925d371d1da9433f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44599 } }
I20260925 09:13:22.404692 3237 leader_election.cc:302] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd67e25423894730925d371d1da9433f; no voters:
Sep 25 09:13:22 dist-test-slave-l6qh krb5kdc[3156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:22.405381 3237 leader_election.cc:288] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:13:22.405462 3242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:22.408747 3237 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:13:22.409576 3242 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [term 1 LEADER]: Becoming Leader. State: Replica: dd67e25423894730925d371d1da9433f, State: Running, Role: LEADER
I20260925 09:13:22.410158 3242 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd67e25423894730925d371d1da9433f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44599 } }
I20260925 09:13:22.413691 3243 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd67e25423894730925d371d1da9433f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd67e25423894730925d371d1da9433f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44599 } } }
I20260925 09:13:22.414618 3243 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:22.415251 3244 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd67e25423894730925d371d1da9433f. Latest consensus state: current_term: 1 leader_uuid: "dd67e25423894730925d371d1da9433f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd67e25423894730925d371d1da9433f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44599 } } }
I20260925 09:13:22.415563 3244 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:22.419111 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:22.372609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.3.19.126:44599)
0925 09:13:22.374268 (+ 1659us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:22.374283 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:22.377108 (+ 2825us) server_negotiation.cc:407] Connection header received
0925 09:13:22.389479 (+ 12371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:22.389511 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:22.389879 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:22.390312 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:22.394038 (+ 3726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:22.395386 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:22.397804 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:22.398575 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:22.398736 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:22.406690 (+ 7954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:22.406707 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:22.406723 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:22.406770 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:22.410784 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:22.411642 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:22.411659 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:22.411665 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:22.411917 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:22.412566 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:22.412574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:22.412579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:22.413537 (+ 958us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:22.413938 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:22.416962 (+ 3024us) server_negotiation.cc:299] Negotiation successful
0925 09:13:22.417485 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1356,"thread_start_us":1026,"threads_started":1}
I20260925 09:13:22.421268 3248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:13:22.427943 3248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:13:22.437882 3248 catalog_manager.cc:1347] Generated new cluster ID: 30922e1481174797bee9cfa5361a03f1
I20260925 09:13:22.437991 3248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:13:22.446784 3248 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:13:22.447984 3248 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:13:22.458405 3248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd67e25423894730925d371d1da9433f: Generated new TSK 0
I20260925 09:13:22.459079 3248 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:13:22.516911 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:44599
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:35583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:22.859354 3263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:22.859612 3263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:22.859704 3263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:22.866669 3263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:22.867131 3263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:22.867277 3263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:13:22.876816 3263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:44599
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:22.878048 3263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:22.879499 3263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:22.892916 3268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:22.892884 3271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:22.892889 3269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:22.893875 3263 server_base.cc:1034] running on GCE node
I20260925 09:13:22.894496 3263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:22.895356 3263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:22.896566 3263 hybrid_clock.cc:648] HybridClock initialized: now 1790327602896535 us; error 57 us; skew 500 ppm
I20260925 09:13:22.896854 3263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:22 dist-test-slave-l6qh krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:22.899854 3263 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:13:22.901293 3263 webserver.cc:460] Webserver started at http://127.3.19.65:46637/ using document root <none> and password file <none>
I20260925 09:13:22.901971 3263 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:22.902113 3263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:22.902400 3263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:22.905646 3263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "93ebe1e836ae45ad93cc7812318c1440"
format_stamp: "Formatted at 2026-09-25 09:13:22 on dist-test-slave-l6qh"
server_key: "956f61d669e9e0b2b38ee4cec60f84a3"
server_key_iv: "7af0ffd15304ef4f9d12908d430e7bc1"
server_key_version: "encryptionkey@0"
I20260925 09:13:22.906386 3263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "93ebe1e836ae45ad93cc7812318c1440"
format_stamp: "Formatted at 2026-09-25 09:13:22 on dist-test-slave-l6qh"
server_key: "956f61d669e9e0b2b38ee4cec60f84a3"
server_key_iv: "7af0ffd15304ef4f9d12908d430e7bc1"
server_key_version: "encryptionkey@0"
I20260925 09:13:22.911264 3263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 09:13:22.914604 3278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:22.916045 3263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:13:22.916249 3263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "93ebe1e836ae45ad93cc7812318c1440"
format_stamp: "Formatted at 2026-09-25 09:13:22 on dist-test-slave-l6qh"
server_key: "956f61d669e9e0b2b38ee4cec60f84a3"
server_key_iv: "7af0ffd15304ef4f9d12908d430e7bc1"
server_key_version: "encryptionkey@0"
I20260925 09:13:22.916435 3263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:22.933337 3263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:22.937952 3263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:22.938244 3263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:22.939155 3263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:22.940515 3263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:22.940622 3263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:22.941147 3263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:22.941228 3263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:22.978466 3263 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:42229
I20260925 09:13:22.978495 3391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:42229 every 8 connection(s)
I20260925 09:13:22.979758 3263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:13:22.985495 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 3263
I20260925 09:13:22.985734 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:13:22.986531 3149 external_mini_cluster.cc:1428] Setting key bf454bfc43c3ca9899a4cee4ec25ae89
Sep 25 09:13:22 dist-test-slave-l6qh krb5kdc[3156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:23.001040 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:22.982251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:40543 (local address 127.3.19.126:44599)
0925 09:13:22.982743 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:22.982749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:22.983772 (+ 1023us) server_negotiation.cc:407] Connection header received
0925 09:13:22.986064 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:22.986068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:22.986135 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:22.986285 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:22.988098 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:22.988918 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:22.990695 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:22.991062 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:22.991189 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:22.995391 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:22.995396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:22.995399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:22.995427 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:22.997928 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:22.998848 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:22.998852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:22.998855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:22.998979 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:22.999463 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:22.999467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:22.999469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:22.999699 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:22.999812 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:23.000660 (+ 848us) server_negotiation.cc:299] Negotiation successful
0925 09:13:23.000812 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":256,"threads_started":1}
I20260925 09:13:23.002559 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:22.982927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:44599 (local address 127.3.19.65:40543)
0925 09:13:22.983619 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:22.983657 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:22.985721 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:22.986560 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:22.986572 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:22.987258 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:22.987940 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:22.987962 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:22.989061 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:22.989066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:22.990534 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:22.990545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:22.991290 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:22.991296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:22.991516 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:22.992565 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:22.992586 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:22.995178 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:22.998053 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:22.998064 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:22.998082 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:22.998720 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:22.999087 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:22.999095 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:22.999102 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:22.999351 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:22.999823 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:22.999834 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:23.000291 (+ 457us) client_negotiation.cc:769] Sending connection context
0925 09:13:23.000589 (+ 298us) client_negotiation.cc:240] Negotiation successful
0925 09:13:23.001076 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":246,"threads_started":1}
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
I20260925 09:13:23.004101 3392 heartbeater.cc:347] Connected to a master server at 127.3.19.126:44599
I20260925 09:13:23.004392 3392 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:23.005154 3392 heartbeater.cc:511] Master 127.3.19.126:44599 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
I20260925 09:13:23.007427 3202 ts_manager.cc:194] Registered new tserver with Master: 93ebe1e836ae45ad93cc7812318c1440 (127.3.19.65:42229)
I20260925 09:13:23.008837 3202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:40543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:13:23.025019 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:44599
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:35583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:23.348107 3398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:23.348363 3398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:23.348526 3398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:23.355257 3398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:23.355383 3398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:23.355537 3398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:13:23.364732 3398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:44599
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:23.365981 3398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:23.367238 3398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:23.380266 3404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:23.380318 3406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:23.380313 3403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:23.380995 3398 server_base.cc:1034] running on GCE node
I20260925 09:13:23.381656 3398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:23.382503 3398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:23.383708 3398 hybrid_clock.cc:648] HybridClock initialized: now 1790327603383652 us; error 49 us; skew 500 ppm
I20260925 09:13:23.384008 3398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:23 dist-test-slave-l6qh krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:23.387022 3398 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:13:23.388432 3398 webserver.cc:460] Webserver started at http://127.3.19.66:46609/ using document root <none> and password file <none>
I20260925 09:13:23.389088 3398 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:23.389266 3398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:23.389592 3398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:23.392293 3398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "7026bc9f2c9641209d98c68ee781367e"
format_stamp: "Formatted at 2026-09-25 09:13:23 on dist-test-slave-l6qh"
server_key: "8d1ad2a3a8dda314701a2887f0e2a42a"
server_key_iv: "165861f99691ee33b4b3fcb1779cbaa0"
server_key_version: "encryptionkey@0"
I20260925 09:13:23.393039 3398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "7026bc9f2c9641209d98c68ee781367e"
format_stamp: "Formatted at 2026-09-25 09:13:23 on dist-test-slave-l6qh"
server_key: "8d1ad2a3a8dda314701a2887f0e2a42a"
server_key_iv: "165861f99691ee33b4b3fcb1779cbaa0"
server_key_version: "encryptionkey@0"
I20260925 09:13:23.398875 3398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 09:13:23.402346 3413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:23.403859 3398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 09:13:23.404021 3398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "7026bc9f2c9641209d98c68ee781367e"
format_stamp: "Formatted at 2026-09-25 09:13:23 on dist-test-slave-l6qh"
server_key: "8d1ad2a3a8dda314701a2887f0e2a42a"
server_key_iv: "165861f99691ee33b4b3fcb1779cbaa0"
server_key_version: "encryptionkey@0"
I20260925 09:13:23.404232 3398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:23.422570 3398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:23.426795 3398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:23.427027 3398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:23.427876 3398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:23.429298 3398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:23.429369 3398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:23.429459 3398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:23.429503 3398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:23.465173 3398 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:45751
I20260925 09:13:23.465267 3525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:45751 every 8 connection(s)
I20260925 09:13:23.466393 3398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:13:23.470247 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 3398
I20260925 09:13:23.470453 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:13:23.470845 3149 external_mini_cluster.cc:1428] Setting key a730f88982f7893e5a3002addac88e00
Sep 25 09:13:23 dist-test-slave-l6qh krb5kdc[3156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
I20260925 09:13:23.486492 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:23.468754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:33803 (local address 127.3.19.126:44599)
0925 09:13:23.468944 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:23.468949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:23.470067 (+ 1118us) server_negotiation.cc:407] Connection header received
0925 09:13:23.471980 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:23.471985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:23.472073 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:23.472166 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:23.473891 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:23.474525 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:23.475979 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:23.476397 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:23.476522 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:23.480306 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:23.480310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:23.480313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:23.480342 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:23.483381 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:23.484280 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:23.484302 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:23.484304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:23.484425 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:23.484881 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:23.484883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:23.484885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:23.485103 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:23.485210 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:23.486079 (+ 869us) server_negotiation.cc:299] Negotiation successful
0925 09:13:23.486266 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 09:13:23.487521 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:23.469195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:44599 (local address 127.3.19.66:33803)
0925 09:13:23.469927 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:23.469965 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:23.471681 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:23.472380 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:23.472388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:23.473206 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:23.473748 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:23.473764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:23.474665 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:23.474669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:23.475845 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:23.475855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:23.476599 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:23.476605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:23.476847 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:23.477490 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:23.477514 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:23.480104 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:23.483527 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:23.483551 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:23.483567 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:23.484171 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:23.484550 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:23.484555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:23.484559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:23.484776 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:23.485225 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:23.485236 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:23.485696 (+ 460us) client_negotiation.cc:769] Sending connection context
0925 09:13:23.486010 (+ 314us) client_negotiation.cc:240] Negotiation successful
0925 09:13:23.486513 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":269,"threads_started":1}
I20260925 09:13:23.489058 3526 heartbeater.cc:347] Connected to a master server at 127.3.19.126:44599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
I20260925 09:13:23.489368 3526 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:23.490170 3526 heartbeater.cc:511] Master 127.3.19.126:44599 requested a full tablet report, sending...
I20260925 09:13:23.491731 3202 ts_manager.cc:194] Registered new tserver with Master: 7026bc9f2c9641209d98c68ee781367e (127.3.19.66:45751)
I20260925 09:13:23.492976 3202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:33803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:13:23.509794 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:44599
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:35583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:23.834887 3531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:23.835180 3531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:23.835338 3531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:23.842589 3531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:23.842762 3531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:23.842938 3531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:13:23.851974 3531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:44599
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:23.853109 3531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:23.854401 3531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:23.867189 3536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:23.867224 3537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:23.867331 3539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:23.867859 3531 server_base.cc:1034] running on GCE node
I20260925 09:13:23.868422 3531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:23.869334 3531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:23.870558 3531 hybrid_clock.cc:648] HybridClock initialized: now 1790327603870511 us; error 52 us; skew 500 ppm
I20260925 09:13:23.870863 3531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:23 dist-test-slave-l6qh krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:23.873883 3531 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:13:23.875252 3531 webserver.cc:460] Webserver started at http://127.3.19.67:42187/ using document root <none> and password file <none>
I20260925 09:13:23.875818 3531 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:23.876282 3531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:23.876569 3531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:23.879201 3531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "d4e34a7255f141a0b0fd882fee8da515"
format_stamp: "Formatted at 2026-09-25 09:13:23 on dist-test-slave-l6qh"
server_key: "4c17dcc0884b051818c90ec2f436f768"
server_key_iv: "18112df683f16961f06687a7427bbb72"
server_key_version: "encryptionkey@0"
I20260925 09:13:23.879866 3531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "d4e34a7255f141a0b0fd882fee8da515"
format_stamp: "Formatted at 2026-09-25 09:13:23 on dist-test-slave-l6qh"
server_key: "4c17dcc0884b051818c90ec2f436f768"
server_key_iv: "18112df683f16961f06687a7427bbb72"
server_key_version: "encryptionkey@0"
I20260925 09:13:23.884742 3531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 09:13:23.888178 3546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:23.889436 3531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:13:23.889555 3531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "d4e34a7255f141a0b0fd882fee8da515"
format_stamp: "Formatted at 2026-09-25 09:13:23 on dist-test-slave-l6qh"
server_key: "4c17dcc0884b051818c90ec2f436f768"
server_key_iv: "18112df683f16961f06687a7427bbb72"
server_key_version: "encryptionkey@0"
I20260925 09:13:23.889714 3531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:23.902555 3531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:23.906875 3531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:23.907119 3531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:23.907917 3531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:23.909261 3531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:23.909349 3531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:23.909464 3531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:23.909508 3531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:23.946149 3531 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:38513
I20260925 09:13:23.946189 3659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:38513 every 8 connection(s)
I20260925 09:13:23.947332 3531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:13:23.953846 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 3531
I20260925 09:13:23.954051 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:13:23.954552 3149 external_mini_cluster.cc:1428] Setting key 663df6eaa2612f3232e324e8de1cdd42
Sep 25 09:13:23 dist-test-slave-l6qh krb5kdc[3156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:23.967024 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:23.949785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:33221 (local address 127.3.19.126:44599)
0925 09:13:23.949989 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:23.949996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:23.951146 (+ 1150us) server_negotiation.cc:407] Connection header received
0925 09:13:23.953080 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:23.953084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:23.953161 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:23.953296 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:23.955078 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:23.956034 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:23.958085 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:23.958504 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:23.958597 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:23.961834 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:23.961839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:23.961842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:23.961872 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:23.964228 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:23.965033 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:23.965038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:23.965040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:23.965220 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:23.965753 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:23.965757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:23.965759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:23.965996 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:23.966101 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:23.966649 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 09:13:23.966792 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 09:13:23.967938 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:23.950270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:44599 (local address 127.3.19.67:33221)
0925 09:13:23.950993 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:23.951034 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:23.952861 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:23.953457 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:23.953464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:23.954184 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:23.954921 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:23.954946 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:23.956164 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:23.956167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:23.957959 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:23.957969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:23.958676 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:23.958696 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:23.958874 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:23.959546 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:23.959566 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:23.961663 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:23.964488 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:23.964494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:23.964504 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:23.964889 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:23.965414 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:23.965418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:23.965420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:23.965617 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:23.966091 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:23.966096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:23.966350 (+ 254us) client_negotiation.cc:769] Sending connection context
0925 09:13:23.966550 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 09:13:23.966924 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":242,"threads_started":1}
I20260925 09:13:23.969280 3660 heartbeater.cc:347] Connected to a master server at 127.3.19.126:44599
I20260925 09:13:23.969591 3660 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:23.970391 3660 heartbeater.cc:511] Master 127.3.19.126:44599 requested a full tablet report, sending...
I20260925 09:13:23.971670 3202 ts_manager.cc:194] Registered new tserver with Master: d4e34a7255f141a0b0fd882fee8da515 (127.3.19.67:38513)
I20260925 09:13:23.972342 3202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:33221
I20260925 09:13:23.982231 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 09:13:23 dist-test-slave-l6qh krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260925 09:13:24.011605 3392 heartbeater.cc:503] Master 127.3.19.126:44599 was elected leader, sending a full tablet report...
Sep 25 09:13:24 dist-test-slave-l6qh krb5kdc[3156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:13:24.025722 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:24.010097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.3.19.65:42229)
0925 09:13:24.010746 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:24.010756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:24.010849 (+ 93us) server_negotiation.cc:407] Connection header received
0925 09:13:24.011031 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:24.011042 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:24.011246 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:24.011396 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:24.012922 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:24.014041 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:24.015194 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:24.015698 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:24.015822 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:24.019981 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:24.019991 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:24.020004 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:24.020033 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:24.023033 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:24.023714 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:24.023721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:24.023723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:24.023835 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:24.024223 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:24.024230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:24.024233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:24.024550 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:24.024843 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:24.025110 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 09:13:24.025400 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":295,"threads_started":1}
W20260925 09:13:24.026928 3306 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47210
Sep 25 09:13:24 dist-test-slave-l6qh krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 09:13:24 dist-test-slave-l6qh krb5kdc[3156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:13:24.069329 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:24.055555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.3.19.65:42229)
0925 09:13:24.055793 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:24.055800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:24.056048 (+ 248us) server_negotiation.cc:407] Connection header received
0925 09:13:24.056262 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:24.056268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:24.056336 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:24.056437 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:24.057835 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:24.058556 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:24.059766 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:24.060187 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:24.060314 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:24.064444 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:24.064449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:24.064452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:24.064483 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:24.066864 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:24.067445 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:24.067452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:24.067456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:24.067574 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:24.067924 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:24.067929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:24.067932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:24.068208 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:24.068446 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:24.068758 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 09:13:24.068971 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 09:13:24.070639 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 3263
I20260925 09:13:24.082459 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:24.071546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.3.19.65:42229)
0925 09:13:24.071736 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:24.071743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:24.071964 (+ 221us) server_negotiation.cc:407] Connection header received
0925 09:13:24.072127 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:24.072131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:24.072181 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:24.072270 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:24.073483 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:24.074123 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:24.075315 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:24.075722 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:24.075862 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:24.077917 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:24.077921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:24.077925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:24.077951 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:24.080104 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:24.080684 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:24.080689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:24.080692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:24.080839 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:24.081163 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:24.081169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:24.081172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:24.081448 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:24.081674 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:24.081965 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 09:13:24.082144 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 09:13:24.083182 3387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:24.164911 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 3263
I20260925 09:13:24.180940 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 3398
Sep 25 09:13:24 dist-test-slave-l6qh krb5kdc[3156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:13:24.196159 3674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:24.181820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.3.19.66:45751)
0925 09:13:24.182331 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:24.182338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:24.182362 (+ 24us) server_negotiation.cc:407] Connection header received
0925 09:13:24.182422 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:24.182428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:24.182624 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:24.182802 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:24.183924 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:24.184909 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:24.186229 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:24.186790 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:24.186924 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:24.190758 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:24.190767 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:24.190780 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:24.190809 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:24.193490 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:24.194111 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:24.194125 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:24.194127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:24.194255 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:24.194622 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:24.194628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:24.194631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:24.194943 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:24.195234 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:24.195555 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 09:13:24.195820 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":236,"threads_started":1}
I20260925 09:13:24.197470 3521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:24.277101 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 3398
I20260925 09:13:24.298727 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 3531
Sep 25 09:13:24 dist-test-slave-l6qh krb5kdc[3156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:13:24.313884 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:24.299844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.3.19.67:38513)
0925 09:13:24.300450 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:24.300457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:24.300485 (+ 28us) server_negotiation.cc:407] Connection header received
0925 09:13:24.300541 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:24.300547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:24.300790 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:24.300913 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:24.302174 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:24.303303 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:24.304498 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:24.304991 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:24.305081 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:24.309038 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:24.309047 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:24.309059 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:24.309088 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:24.311633 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:24.312203 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:24.312209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:24.312211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:24.312339 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:24.312614 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:24.312617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:24.312619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:24.312893 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:24.313115 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:24.313420 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 09:13:24.313613 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":328,"threads_started":1}
I20260925 09:13:24.315060 3655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:24.393544 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 3531
I20260925 09:13:24.410243 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 3169
I20260925 09:13:24.411404 3232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:24.509533 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 3169
2026-09-25T09:13:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4018 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:13:24 dist-test-slave-l6qh krb5kdc[3681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:13:24 dist-test-slave-l6qh krb5kdc[3681](info): set up 2 sockets
Sep 25 09:13:24 dist-test-slave-l6qh krb5kdc[3681](info): commencing operation
krb5kdc: starting...
W20260925 09:13:25.586597 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:13:25 dist-test-slave-l6qh krb5kdc[3681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:13:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:13:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:13:25.768980 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33799
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:36187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:33799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:26.080793 3694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:26.081113 3694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:26.081295 3694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:26.088361 3694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:13:26.088497 3694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:26.088603 3694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:26.088696 3694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:13:26.088774 3694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:13:26.098302 3694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:36187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:33799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:26.099601 3694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:26.100867 3694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:26.112689 3699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:26.112619 3700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:26.112660 3702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:26.113490 3694 server_base.cc:1034] running on GCE node
I20260925 09:13:26.114287 3694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:26.115640 3694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:26.116868 3694 hybrid_clock.cc:648] HybridClock initialized: now 1790327606116823 us; error 54 us; skew 500 ppm
I20260925 09:13:26.117220 3694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:26 dist-test-slave-l6qh krb5kdc[3681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327606, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:26.125605 3694 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:13:26.127004 3694 webserver.cc:460] Webserver started at http://127.3.19.126:39419/ using document root <none> and password file <none>
I20260925 09:13:26.127681 3694 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:26.127823 3694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:26.128122 3694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:26.130983 3694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "f3727d4e94aa4f418e9aaf8d79b3ebd3"
format_stamp: "Formatted at 2026-09-25 09:13:26 on dist-test-slave-l6qh"
server_key: "15803c1e12c5a9e22a6261c378e863bc"
server_key_iv: "b4527850898707742cd1d392664d0e8c"
server_key_version: "encryptionkey@0"
I20260925 09:13:26.131690 3694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "f3727d4e94aa4f418e9aaf8d79b3ebd3"
format_stamp: "Formatted at 2026-09-25 09:13:26 on dist-test-slave-l6qh"
server_key: "15803c1e12c5a9e22a6261c378e863bc"
server_key_iv: "b4527850898707742cd1d392664d0e8c"
server_key_version: "encryptionkey@0"
I20260925 09:13:26.136443 3694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 09:13:26.139725 3709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:26.140889 3694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 09:13:26.141055 3694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "f3727d4e94aa4f418e9aaf8d79b3ebd3"
format_stamp: "Formatted at 2026-09-25 09:13:26 on dist-test-slave-l6qh"
server_key: "15803c1e12c5a9e22a6261c378e863bc"
server_key_iv: "b4527850898707742cd1d392664d0e8c"
server_key_version: "encryptionkey@0"
I20260925 09:13:26.141252 3694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:26.163563 3694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:26.167464 3694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:26.167723 3694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:26.186857 3694 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:33799
I20260925 09:13:26.186880 3761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:33799 every 8 connection(s)
I20260925 09:13:26.188259 3694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:13:26.191824 3762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:26.195140 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 3694
I20260925 09:13:26.195379 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:13:26.196179 3149 external_mini_cluster.cc:1428] Setting key 3faa163438ef83c800484be952c24996
I20260925 09:13:26.204690 3762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3: Bootstrap starting.
I20260925 09:13:26.208354 3762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:26.209649 3762 log.cc:824] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3: Log is configured to *not* fsync() on all Append() calls
Sep 25 09:13:26 dist-test-slave-l6qh krb5kdc[3681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:26.212342 3762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3: No bootstrap required, opened a new log
I20260925 09:13:26.218698 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:26.198292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.3.19.126:33799)
0925 09:13:26.199100 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:26.199112 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:26.199157 (+ 45us) server_negotiation.cc:407] Connection header received
0925 09:13:26.200790 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:26.200812 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:26.201218 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:26.201632 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:26.203288 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:26.204621 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:26.206462 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:26.207133 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:26.207278 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:26.211807 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:26.211814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:26.211827 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:26.211862 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:26.214860 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:26.215562 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:26.215568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:26.215576 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:26.215732 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:26.216098 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:26.216101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:26.216103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:26.216561 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:26.216865 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:26.217373 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 09:13:26.217772 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":396,"threads_started":1}
I20260925 09:13:26.220180 3762 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3727d4e94aa4f418e9aaf8d79b3ebd3" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33799 } }
I20260925 09:13:26.220544 3762 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:26.220649 3762 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3727d4e94aa4f418e9aaf8d79b3ebd3, State: Initialized, Role: FOLLOWER
I20260925 09:13:26.221203 3762 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3727d4e94aa4f418e9aaf8d79b3ebd3" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33799 } }
I20260925 09:13:26.221521 3762 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:13:26.221661 3762 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:13:26.221815 3762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:26.223796 3762 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3727d4e94aa4f418e9aaf8d79b3ebd3" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33799 } }
I20260925 09:13:26.224308 3762 leader_election.cc:302] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3727d4e94aa4f418e9aaf8d79b3ebd3; no voters:
I20260925 09:13:26.224925 3762 leader_election.cc:288] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:13:26.225024 3767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:26.225678 3767 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [term 1 LEADER]: Becoming Leader. State: Replica: f3727d4e94aa4f418e9aaf8d79b3ebd3, State: Running, Role: LEADER
I20260925 09:13:26.226212 3767 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3727d4e94aa4f418e9aaf8d79b3ebd3" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33799 } }
I20260925 09:13:26.227037 3762 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:13:26.228875 3768 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3727d4e94aa4f418e9aaf8d79b3ebd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3727d4e94aa4f418e9aaf8d79b3ebd3" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33799 } } }
I20260925 09:13:26.229393 3768 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:26.228938 3769 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3727d4e94aa4f418e9aaf8d79b3ebd3. Latest consensus state: current_term: 1 leader_uuid: "f3727d4e94aa4f418e9aaf8d79b3ebd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3727d4e94aa4f418e9aaf8d79b3ebd3" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33799 } } }
I20260925 09:13:26.233911 3769 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3 [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:26.234519 3773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:13:26.238574 3773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:13:26.246665 3773 catalog_manager.cc:1347] Generated new cluster ID: ae03c50ec53c45b28638b9d96b132e30
I20260925 09:13:26.246783 3773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:13:26.255861 3773 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:13:26.257047 3773 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:13:26.265143 3773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3727d4e94aa4f418e9aaf8d79b3ebd3: Generated new TSK 0
I20260925 09:13:26.265915 3773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:13:26.312891 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33799
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:36187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:26.640924 3788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:26.641184 3788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:26.641336 3788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:26.648386 3788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:26.648478 3788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:26.648592 3788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:13:26.657845 3788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:36187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:26.658949 3788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:26.660133 3788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:26.672639 3793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:26.672708 3794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:26.672730 3796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:26.673434 3788 server_base.cc:1034] running on GCE node
I20260925 09:13:26.673884 3788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:26.674613 3788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:26.675792 3788 hybrid_clock.cc:648] HybridClock initialized: now 1790327606675754 us; error 29 us; skew 500 ppm
I20260925 09:13:26.676118 3788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:26 dist-test-slave-l6qh krb5kdc[3681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327606, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:26.679071 3788 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:13:26.680320 3788 webserver.cc:460] Webserver started at http://127.3.19.65:38257/ using document root <none> and password file <none>
I20260925 09:13:26.680881 3788 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:26.680974 3788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:26.681206 3788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:26.683780 3788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "aae3229168d745ba9e2dd6acf3948cd8"
format_stamp: "Formatted at 2026-09-25 09:13:26 on dist-test-slave-l6qh"
server_key: "f0273718d973442dbefa5326e76f97e5"
server_key_iv: "623fb77c6ce00c8f2750f7aa122be70e"
server_key_version: "encryptionkey@0"
I20260925 09:13:26.684532 3788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "aae3229168d745ba9e2dd6acf3948cd8"
format_stamp: "Formatted at 2026-09-25 09:13:26 on dist-test-slave-l6qh"
server_key: "f0273718d973442dbefa5326e76f97e5"
server_key_iv: "623fb77c6ce00c8f2750f7aa122be70e"
server_key_version: "encryptionkey@0"
I20260925 09:13:26.689577 3788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 09:13:26.694177 3803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:26.695804 3788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260925 09:13:26.695959 3788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "aae3229168d745ba9e2dd6acf3948cd8"
format_stamp: "Formatted at 2026-09-25 09:13:26 on dist-test-slave-l6qh"
server_key: "f0273718d973442dbefa5326e76f97e5"
server_key_iv: "623fb77c6ce00c8f2750f7aa122be70e"
server_key_version: "encryptionkey@0"
I20260925 09:13:26.696147 3788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:26.715590 3788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:26.720130 3788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:26.720424 3788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:26.721346 3788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:26.722604 3788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:26.722702 3788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:26.722839 3788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:26.722915 3788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:26.753337 3788 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:45839
I20260925 09:13:26.753381 3915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:45839 every 8 connection(s)
I20260925 09:13:26.754511 3788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:13:26.757097 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 3788
I20260925 09:13:26.757328 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:13:26.758101 3149 external_mini_cluster.cc:1428] Setting key da0d1d32f3596e0794d0790ccd45bdcf
Sep 25 09:13:26 dist-test-slave-l6qh krb5kdc[3681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327606, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
I20260925 09:13:26.774308 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:26.756983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:34457 (local address 127.3.19.126:33799)
0925 09:13:26.757406 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:26.757412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:26.758673 (+ 1261us) server_negotiation.cc:407] Connection header received
0925 09:13:26.760648 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:26.760653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:26.760739 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:26.760856 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:26.762314 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:26.763010 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:26.764902 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:26.765343 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:26.765456 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:26.768948 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:26.768953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:26.768956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:26.768986 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:26.771432 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:26.772096 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:26.772103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:26.772106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:26.772232 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:26.772614 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:26.772620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:26.772623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:26.772904 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:26.773038 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:26.773745 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 09:13:26.773961 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":206,"threads_started":1}
I20260925 09:13:26.775234 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:26.757703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33799 (local address 127.3.19.65:34457)
0925 09:13:26.758524 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:26.758560 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:26.760412 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:26.761078 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:26.761091 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:26.761630 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:26.762151 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:26.762167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:26.763150 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:26.763155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:26.764767 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:26.764784 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:26.765569 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:26.765574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:26.765763 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:26.766514 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:26.766538 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:26.768751 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:26.771561 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:26.771570 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:26.771585 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:26.771987 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:26.772341 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:26.772346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:26.772349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:26.772513 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:26.773027 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:26.773034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:26.773388 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 09:13:26.773650 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 09:13:26.774140 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":332,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
I20260925 09:13:26.776798 3916 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33799
I20260925 09:13:26.777096 3916 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:26.777796 3916 heartbeater.cc:511] Master 127.3.19.126:33799 requested a full tablet report, sending...
I20260925 09:13:26.779966 3726 ts_manager.cc:194] Registered new tserver with Master: aae3229168d745ba9e2dd6acf3948cd8 (127.3.19.65:45839)
I20260925 09:13:26.781502 3726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:34457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:13:26.797796 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33799
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:36187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:27.112982 3922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:27.113245 3922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:27.113416 3922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:27.120231 3922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:27.120318 3922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:27.120510 3922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:13:27.130005 3922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:36187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:27.131138 3922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:27.132498 3922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:27.144788 3928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:27.144804 3931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:27.144848 3929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:27.145884 3922 server_base.cc:1034] running on GCE node
I20260925 09:13:27.146410 3922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:27.147222 3922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:27.148432 3922 hybrid_clock.cc:648] HybridClock initialized: now 1790327607148372 us; error 50 us; skew 500 ppm
I20260925 09:13:27.148721 3922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:27 dist-test-slave-l6qh krb5kdc[3681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:27.151824 3922 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:13:27.153112 3922 webserver.cc:460] Webserver started at http://127.3.19.66:37715/ using document root <none> and password file <none>
I20260925 09:13:27.153788 3922 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:27.153916 3922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:27.154186 3922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:27.156781 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "44a0e9c0a8874ae9b5e793b06335c7df"
format_stamp: "Formatted at 2026-09-25 09:13:27 on dist-test-slave-l6qh"
server_key: "6ad908312fca4f19a0bffc9c406589bb"
server_key_iv: "e5ffe03e0fb5c4ace15f814448d15ab5"
server_key_version: "encryptionkey@0"
I20260925 09:13:27.157537 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "44a0e9c0a8874ae9b5e793b06335c7df"
format_stamp: "Formatted at 2026-09-25 09:13:27 on dist-test-slave-l6qh"
server_key: "6ad908312fca4f19a0bffc9c406589bb"
server_key_iv: "e5ffe03e0fb5c4ace15f814448d15ab5"
server_key_version: "encryptionkey@0"
I20260925 09:13:27.162323 3922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 09:13:27.165486 3938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:27.166601 3922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 09:13:27.166759 3922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "44a0e9c0a8874ae9b5e793b06335c7df"
format_stamp: "Formatted at 2026-09-25 09:13:27 on dist-test-slave-l6qh"
server_key: "6ad908312fca4f19a0bffc9c406589bb"
server_key_iv: "e5ffe03e0fb5c4ace15f814448d15ab5"
server_key_version: "encryptionkey@0"
I20260925 09:13:27.166952 3922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:27.182058 3922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:27.185979 3922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:27.186264 3922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:27.187227 3922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:27.188500 3922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:27.188601 3922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:27.188740 3922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:27.188817 3922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:27.221664 3922 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:43735
I20260925 09:13:27.221709 4050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:43735 every 8 connection(s)
I20260925 09:13:27.222909 3922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:13:27.232833 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 3922
I20260925 09:13:27.233021 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:13:27.233479 3149 external_mini_cluster.cc:1428] Setting key 40f3221b05e065338a95d6b66a4fa391
Sep 25 09:13:27 dist-test-slave-l6qh krb5kdc[3681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:27.243445 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.225357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:34361 (local address 127.3.19.126:33799)
0925 09:13:27.225519 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:27.225523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:27.226649 (+ 1126us) server_negotiation.cc:407] Connection header received
0925 09:13:27.228606 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:27.228612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:27.228699 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:27.228816 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:27.230278 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.231074 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.232693 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.233201 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.233364 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.236959 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:27.236967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:27.236975 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:27.237017 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:27.240438 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:27.241334 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:27.241345 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:27.241349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:27.241473 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:27.241939 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:27.241942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:27.241944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:27.242167 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:27.242267 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:27.243073 (+ 806us) server_negotiation.cc:299] Negotiation successful
0925 09:13:27.243224 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 09:13:27.245005 4053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.225827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33799 (local address 127.3.19.66:34361)
0925 09:13:27.226497 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:27.226535 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:27.228353 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:27.229046 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:27.229054 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:27.229611 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:27.230111 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.230125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.231262 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.231272 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.232551 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.232562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.233447 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.233456 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.233744 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.234412 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:27.234474 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:27.236742 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:27.240568 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:27.240578 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:27.240594 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:27.241205 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:27.241582 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:27.241589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:27.241595 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:27.241833 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:27.242280 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:27.242291 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:27.242705 (+ 414us) client_negotiation.cc:769] Sending connection context
0925 09:13:27.242997 (+ 292us) client_negotiation.cc:240] Negotiation successful
0925 09:13:27.243531 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":247,"threads_started":1}
I20260925 09:13:27.246409 4051 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33799
I20260925 09:13:27.246690 4051 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:27.247412 4051 heartbeater.cc:511] Master 127.3.19.126:33799 requested a full tablet report, sending...
I20260925 09:13:27.248771 3726 ts_manager.cc:194] Registered new tserver with Master: 44a0e9c0a8874ae9b5e793b06335c7df (127.3.19.66:43735)
I20260925 09:13:27.249603 3726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:34361
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:13:27.273583 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33799
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:36187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:27.584007 4056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:27.584352 4056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:27.584537 4056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:27.591919 4056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:27.592083 4056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:27.592293 4056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:13:27.602353 4056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:36187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:27.603662 4056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:27.605077 4056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:27.618631 4064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:27.618700 4061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:27.618724 4062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:27.619422 4056 server_base.cc:1034] running on GCE node
I20260925 09:13:27.620025 4056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:27.620934 4056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:27.622164 4056 hybrid_clock.cc:648] HybridClock initialized: now 1790327607622110 us; error 39 us; skew 500 ppm
I20260925 09:13:27.622527 4056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:27 dist-test-slave-l6qh krb5kdc[3681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:27.625950 4056 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:13:27.627396 4056 webserver.cc:460] Webserver started at http://127.3.19.67:44801/ using document root <none> and password file <none>
I20260925 09:13:27.628026 4056 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:27.628132 4056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:27.628453 4056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:27.631462 4056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "a07f14cdf85a49f7ab01fb0c1b57001d"
format_stamp: "Formatted at 2026-09-25 09:13:27 on dist-test-slave-l6qh"
server_key: "b99b6c65574e00a373b7c5c627cbf5cd"
server_key_iv: "5da0b5ae86244bbdc2ded8c5c3d4e407"
server_key_version: "encryptionkey@0"
I20260925 09:13:27.632318 4056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "a07f14cdf85a49f7ab01fb0c1b57001d"
format_stamp: "Formatted at 2026-09-25 09:13:27 on dist-test-slave-l6qh"
server_key: "b99b6c65574e00a373b7c5c627cbf5cd"
server_key_iv: "5da0b5ae86244bbdc2ded8c5c3d4e407"
server_key_version: "encryptionkey@0"
I20260925 09:13:27.637305 4056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 09:13:27.640774 4071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:27.642189 4056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:13:27.642364 4056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "a07f14cdf85a49f7ab01fb0c1b57001d"
format_stamp: "Formatted at 2026-09-25 09:13:27 on dist-test-slave-l6qh"
server_key: "b99b6c65574e00a373b7c5c627cbf5cd"
server_key_iv: "5da0b5ae86244bbdc2ded8c5c3d4e407"
server_key_version: "encryptionkey@0"
I20260925 09:13:27.642588 4056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:27.675246 4056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:27.679668 4056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:27.679960 4056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:27.680964 4056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:27.682367 4056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:27.682473 4056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:27.682616 4056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:27.682684 4056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:27.713326 4056 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:41355
I20260925 09:13:27.713366 4184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:41355 every 8 connection(s)
I20260925 09:13:27.714603 4056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:13:27.721088 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 4056
I20260925 09:13:27.721306 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:13:27.721764 3149 external_mini_cluster.cc:1428] Setting key 93b1464f7d642a89599defec0de1dfe7
Sep 25 09:13:27 dist-test-slave-l6qh krb5kdc[3681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:27.734813 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.717055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:38253 (local address 127.3.19.126:33799)
0925 09:13:27.717267 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:27.717276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:27.718496 (+ 1220us) server_negotiation.cc:407] Connection header received
0925 09:13:27.720403 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:27.720409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:27.720504 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:27.720623 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:27.722307 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.723128 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.724892 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.725376 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.725533 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.729098 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:27.729109 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:27.729112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:27.729144 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:27.731782 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:27.732470 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:27.732478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:27.732484 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:27.732621 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:27.733049 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:27.733054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:27.733057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:27.733402 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:27.733557 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:27.734229 (+ 672us) server_negotiation.cc:299] Negotiation successful
0925 09:13:27.734439 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 09:13:27.735809 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.717570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33799 (local address 127.3.19.67:38253)
0925 09:13:27.718317 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:27.718362 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:27.720135 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:27.720856 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:27.720864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:27.721517 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:27.722138 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.722156 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.723300 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.723305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.724748 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.724760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.725608 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.725617 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.725914 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.726618 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:27.726640 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:27.728900 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:27.731925 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:27.731935 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:27.731949 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:27.732348 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:27.732736 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:27.732742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:27.732746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:27.732923 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:27.733530 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:27.733538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:27.733852 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 09:13:27.734104 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 09:13:27.734559 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":300,"threads_started":1}
I20260925 09:13:27.737062 4185 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33799
I20260925 09:13:27.737399 4185 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:27.738065 4185 heartbeater.cc:511] Master 127.3.19.126:33799 requested a full tablet report, sending...
I20260925 09:13:27.739360 3726 ts_manager.cc:194] Registered new tserver with Master: a07f14cdf85a49f7ab01fb0c1b57001d (127.3.19.67:41355)
I20260925 09:13:27.740293 3726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:38253
I20260925 09:13:27.748373 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:13:27.767247 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.754938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58052 (local address 127.3.19.126:33799)
0925 09:13:27.755116 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:27.755123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:27.755664 (+ 541us) server_negotiation.cc:407] Connection header received
0925 09:13:27.755838 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:27.755844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:27.755908 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:27.756002 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:27.757417 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.758130 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.759425 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.759837 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.759982 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.762142 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:27.762149 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:27.762158 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:27.762190 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:27.764312 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:27.764917 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:27.764924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:27.764927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:27.765059 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:27.765468 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:27.765475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:27.765480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:27.765758 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:27.766161 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:27.766356 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 09:13:27.766557 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 09:13:27.773029 3726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:13:27.780069 3726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:13:27.784330 3916 heartbeater.cc:503] Master 127.3.19.126:33799 was elected leader, sending a full tablet report...
I20260925 09:13:27.826579 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.792523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.3.19.66:43735)
0925 09:13:27.817444 (+ 24921us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:27.817457 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:27.817502 (+ 45us) server_negotiation.cc:407] Connection header received
0925 09:13:27.817812 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:27.817825 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:27.818189 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:27.818425 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:27.819521 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.820849 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.823749 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.825208 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.825372 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.825613 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:27.826018 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 09:13:27.826183 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24687,"thread_start_us":25564,"threads_started":1}
I20260925 09:13:27.826612 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.791966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:43735 (local address 127.0.0.1:32946)
0925 09:13:27.817406 (+ 25440us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:27.817427 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:27.817551 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:27.818745 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:27.818756 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:27.818796 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:27.819342 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.819354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.821013 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.821018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.823570 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.823584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.825444 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.825450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.825655 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.825806 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 09:13:27.825960 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 09:13:27.826148 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25262,"mutex_wait_us":21,"thread_start_us":23832,"threads_started":1}
I20260925 09:13:27.827688 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.791471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:45839 (local address 127.0.0.1:35228)
0925 09:13:27.816358 (+ 24887us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:27.816439 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:27.816705 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:27.820247 (+ 3542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:27.820252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:27.820278 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:27.820600 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.820610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.822690 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.822698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.824794 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.824808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.826850 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.826856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.826998 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.827028 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 09:13:27.827125 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 09:13:27.827257 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24525,"mutex_wait_us":53,"thread_start_us":26959,"threads_started":1}
I20260925 09:13:27.828038 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.792064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:41355 (local address 127.0.0.1:47804)
0925 09:13:27.818715 (+ 26651us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:27.818729 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:27.818872 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:27.819489 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:27.819494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:27.819518 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:27.819971 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.819984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.822838 (+ 2854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.822848 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.824976 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.824986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.827125 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.827132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.827318 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.827343 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 09:13:27.827406 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 09:13:27.827562 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26553,"mutex_wait_us":19,"thread_start_us":26947,"threads_started":1}
I20260925 09:13:27.829094 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.792016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.3.19.65:45839)
0925 09:13:27.819659 (+ 27643us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:27.819666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:27.819691 (+ 25us) server_negotiation.cc:407] Connection header received
0925 09:13:27.819781 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:27.819789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:27.820043 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:27.820216 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:27.820919 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.822573 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.824983 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.826337 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.826452 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.826696 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:27.828629 (+ 1933us) server_negotiation.cc:299] Negotiation successful
0925 09:13:27.828784 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27459,"thread_start_us":27295,"threads_started":1}
I20260925 09:13:27.829312 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.792753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.3.19.67:41355)
0925 09:13:27.815767 (+ 23014us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:27.815779 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:27.818830 (+ 3051us) server_negotiation.cc:407] Connection header received
0925 09:13:27.819044 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:27.819051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:27.819273 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:27.820164 (+ 891us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:27.820675 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.822314 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.825134 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.826869 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.827807 (+ 938us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.828262 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:27.828346 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 09:13:27.828520 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22775,"thread_start_us":22594,"threads_started":1}
I20260925 09:13:27.830989 3986 tablet_service.cc:1467] Processing CreateTablet for tablet e032c2cd70f54ec0afa71ea2c5e3eb04 (DEFAULT_TABLE table=test-table [id=64f447fba5764667ad8c711547818ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:27.832039 3851 tablet_service.cc:1467] Processing CreateTablet for tablet e032c2cd70f54ec0afa71ea2c5e3eb04 (DEFAULT_TABLE table=test-table [id=64f447fba5764667ad8c711547818ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:27.832496 3986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e032c2cd70f54ec0afa71ea2c5e3eb04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:27.832705 4120 tablet_service.cc:1467] Processing CreateTablet for tablet e032c2cd70f54ec0afa71ea2c5e3eb04 (DEFAULT_TABLE table=test-table [id=64f447fba5764667ad8c711547818ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:27.833842 3851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e032c2cd70f54ec0afa71ea2c5e3eb04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:27.834060 4120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e032c2cd70f54ec0afa71ea2c5e3eb04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:27.845119 4203 tablet_bootstrap.cc:492] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8: Bootstrap starting.
I20260925 09:13:27.848224 4205 tablet_bootstrap.cc:492] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d: Bootstrap starting.
I20260925 09:13:27.849844 4204 tablet_bootstrap.cc:492] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df: Bootstrap starting.
I20260925 09:13:27.850448 4203 tablet_bootstrap.cc:654] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:27.851795 4203 log.cc:824] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:27.852517 4205 tablet_bootstrap.cc:654] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:27.852686 4204 tablet_bootstrap.cc:654] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:27.853734 4204 log.cc:824] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:27.853749 4205 log.cc:824] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:27.854979 4203 tablet_bootstrap.cc:492] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8: No bootstrap required, opened a new log
I20260925 09:13:27.855271 4203 ts_tablet_manager.cc:1397] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 09:13:27.856685 4204 tablet_bootstrap.cc:492] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df: No bootstrap required, opened a new log
I20260925 09:13:27.856720 4205 tablet_bootstrap.cc:492] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d: No bootstrap required, opened a new log
I20260925 09:13:27.856928 4204 ts_tablet_manager.cc:1397] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260925 09:13:27.856940 4205 ts_tablet_manager.cc:1397] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 09:13:27.864010 4205 raft_consensus.cc:348] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae3229168d745ba9e2dd6acf3948cd8" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45839 } } peers { permanent_uuid: "a07f14cdf85a49f7ab01fb0c1b57001d" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41355 } } peers { permanent_uuid: "44a0e9c0a8874ae9b5e793b06335c7df" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43735 } }
I20260925 09:13:27.864341 4205 raft_consensus.cc:374] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:27.864420 4205 raft_consensus.cc:729] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07f14cdf85a49f7ab01fb0c1b57001d, State: Initialized, Role: FOLLOWER
I20260925 09:13:27.864976 4205 consensus_queue.cc:260] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae3229168d745ba9e2dd6acf3948cd8" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45839 } } peers { permanent_uuid: "a07f14cdf85a49f7ab01fb0c1b57001d" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41355 } } peers { permanent_uuid: "44a0e9c0a8874ae9b5e793b06335c7df" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43735 } }
I20260925 09:13:27.866035 4185 heartbeater.cc:503] Master 127.3.19.126:33799 was elected leader, sending a full tablet report...
I20260925 09:13:27.866658 4205 ts_tablet_manager.cc:1428] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260925 09:13:27.867062 4203 raft_consensus.cc:348] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae3229168d745ba9e2dd6acf3948cd8" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45839 } } peers { permanent_uuid: "a07f14cdf85a49f7ab01fb0c1b57001d" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41355 } } peers { permanent_uuid: "44a0e9c0a8874ae9b5e793b06335c7df" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43735 } }
I20260925 09:13:27.867575 4204 raft_consensus.cc:348] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae3229168d745ba9e2dd6acf3948cd8" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45839 } } peers { permanent_uuid: "a07f14cdf85a49f7ab01fb0c1b57001d" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41355 } } peers { permanent_uuid: "44a0e9c0a8874ae9b5e793b06335c7df" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43735 } }
I20260925 09:13:27.867972 4204 raft_consensus.cc:374] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:27.868079 4204 raft_consensus.cc:729] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a0e9c0a8874ae9b5e793b06335c7df, State: Initialized, Role: FOLLOWER
I20260925 09:13:27.868122 4203 raft_consensus.cc:374] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:27.868328 4203 raft_consensus.cc:729] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae3229168d745ba9e2dd6acf3948cd8, State: Initialized, Role: FOLLOWER
I20260925 09:13:27.868739 4204 consensus_queue.cc:260] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae3229168d745ba9e2dd6acf3948cd8" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45839 } } peers { permanent_uuid: "a07f14cdf85a49f7ab01fb0c1b57001d" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41355 } } peers { permanent_uuid: "44a0e9c0a8874ae9b5e793b06335c7df" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43735 } }
I20260925 09:13:27.868958 4203 consensus_queue.cc:260] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae3229168d745ba9e2dd6acf3948cd8" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45839 } } peers { permanent_uuid: "a07f14cdf85a49f7ab01fb0c1b57001d" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41355 } } peers { permanent_uuid: "44a0e9c0a8874ae9b5e793b06335c7df" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43735 } }
I20260925 09:13:27.870117 4051 heartbeater.cc:503] Master 127.3.19.126:33799 was elected leader, sending a full tablet report...
I20260925 09:13:27.870602 4204 ts_tablet_manager.cc:1428] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260925 09:13:27.872614 4203 ts_tablet_manager.cc:1428] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260925 09:13:27.900194 4211 raft_consensus.cc:482] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:13:27.900480 4211 raft_consensus.cc:504] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae3229168d745ba9e2dd6acf3948cd8" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45839 } } peers { permanent_uuid: "a07f14cdf85a49f7ab01fb0c1b57001d" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41355 } } peers { permanent_uuid: "44a0e9c0a8874ae9b5e793b06335c7df" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43735 } }
I20260925 09:13:27.902235 4211 leader_election.cc:288] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a07f14cdf85a49f7ab01fb0c1b57001d (127.3.19.67:41355), 44a0e9c0a8874ae9b5e793b06335c7df (127.3.19.66:43735)
I20260925 09:13:27.908792 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.902616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:37153 (local address 127.3.19.67:41355)
0925 09:13:27.902829 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:27.902835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:27.903475 (+ 640us) server_negotiation.cc:407] Connection header received
0925 09:13:27.903682 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:27.903688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:27.903780 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:27.903844 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:27.904485 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.905265 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.907055 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.907811 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.907901 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.908002 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:27.908392 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 09:13:27.908522 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 09:13:27.908948 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.902596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:41355 (local address 127.3.19.65:37153)
0925 09:13:27.903350 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:27.903370 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:27.903500 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:27.903898 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:27.903904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:27.903933 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:27.904339 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.904348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.905398 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.905403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.906907 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.906917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.908018 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.908024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.908245 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.908268 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 09:13:27.908340 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 09:13:27.908448 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":4,"thread_start_us":394,"threads_started":1}
I20260925 09:13:27.908996 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.902745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:36385 (local address 127.3.19.66:43735)
0925 09:13:27.902958 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:27.902967 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:27.903586 (+ 619us) server_negotiation.cc:407] Connection header received
0925 09:13:27.903785 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:27.903789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:27.903877 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:27.903964 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:27.904612 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.905413 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.907132 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.907882 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.908016 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.908121 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:27.908496 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 09:13:27.908659 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 09:13:27.909333 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.902695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:43735 (local address 127.3.19.65:36385)
0925 09:13:27.903477 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:27.903494 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:27.903640 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:27.904003 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:27.904006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:27.904035 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:27.904439 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.904453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.905537 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.905541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.906964 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:27.907001 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.908127 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.908132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:27.908302 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.908355 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 09:13:27.908446 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 09:13:27.908557 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":42,"thread_start_us":460,"threads_started":1}
I20260925 09:13:27.910172 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e032c2cd70f54ec0afa71ea2c5e3eb04" candidate_uuid: "aae3229168d745ba9e2dd6acf3948cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a0e9c0a8874ae9b5e793b06335c7df" is_pre_election: true
I20260925 09:13:27.910447 4140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e032c2cd70f54ec0afa71ea2c5e3eb04" candidate_uuid: "aae3229168d745ba9e2dd6acf3948cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07f14cdf85a49f7ab01fb0c1b57001d" is_pre_election: true
I20260925 09:13:27.910619 4006 raft_consensus.cc:2454] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae3229168d745ba9e2dd6acf3948cd8 in term 0.
I20260925 09:13:27.910962 4140 raft_consensus.cc:2454] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae3229168d745ba9e2dd6acf3948cd8 in term 0.
I20260925 09:13:27.911278 3805 leader_election.cc:302] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a0e9c0a8874ae9b5e793b06335c7df, aae3229168d745ba9e2dd6acf3948cd8; no voters:
I20260925 09:13:27.911666 4211 raft_consensus.cc:2781] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:13:27.911819 4211 raft_consensus.cc:482] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:13:27.911929 4211 raft_consensus.cc:3037] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:27.914134 4211 raft_consensus.cc:504] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae3229168d745ba9e2dd6acf3948cd8" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45839 } } peers { permanent_uuid: "a07f14cdf85a49f7ab01fb0c1b57001d" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41355 } } peers { permanent_uuid: "44a0e9c0a8874ae9b5e793b06335c7df" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43735 } }
I20260925 09:13:27.914661 4211 leader_election.cc:288] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [CANDIDATE]: Term 1 election: Requested vote from peers a07f14cdf85a49f7ab01fb0c1b57001d (127.3.19.67:41355), 44a0e9c0a8874ae9b5e793b06335c7df (127.3.19.66:43735)
I20260925 09:13:27.915084 4140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e032c2cd70f54ec0afa71ea2c5e3eb04" candidate_uuid: "aae3229168d745ba9e2dd6acf3948cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07f14cdf85a49f7ab01fb0c1b57001d"
I20260925 09:13:27.915103 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e032c2cd70f54ec0afa71ea2c5e3eb04" candidate_uuid: "aae3229168d745ba9e2dd6acf3948cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a0e9c0a8874ae9b5e793b06335c7df"
I20260925 09:13:27.915278 4140 raft_consensus.cc:3037] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:27.915285 4006 raft_consensus.cc:3037] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:27.918422 4006 raft_consensus.cc:2454] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae3229168d745ba9e2dd6acf3948cd8 in term 1.
I20260925 09:13:27.918428 4140 raft_consensus.cc:2454] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae3229168d745ba9e2dd6acf3948cd8 in term 1.
I20260925 09:13:27.918794 3805 leader_election.cc:302] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a0e9c0a8874ae9b5e793b06335c7df, aae3229168d745ba9e2dd6acf3948cd8; no voters:
I20260925 09:13:27.919073 4211 raft_consensus.cc:2781] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:27.919515 4211 raft_consensus.cc:686] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [term 1 LEADER]: Becoming Leader. State: Replica: aae3229168d745ba9e2dd6acf3948cd8, State: Running, Role: LEADER
I20260925 09:13:27.919970 4211 consensus_queue.cc:237] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae3229168d745ba9e2dd6acf3948cd8" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45839 } } peers { permanent_uuid: "a07f14cdf85a49f7ab01fb0c1b57001d" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41355 } } peers { permanent_uuid: "44a0e9c0a8874ae9b5e793b06335c7df" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43735 } }
I20260925 09:13:27.924937 3725 catalog_manager.cc:5495] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae3229168d745ba9e2dd6acf3948cd8 (127.3.19.65). New cstate: current_term: 1 leader_uuid: "aae3229168d745ba9e2dd6acf3948cd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae3229168d745ba9e2dd6acf3948cd8" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a07f14cdf85a49f7ab01fb0c1b57001d" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44a0e9c0a8874ae9b5e793b06335c7df" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43735 } health_report { overall_health: UNKNOWN } } }
Sep 25 09:13:27 dist-test-slave-l6qh krb5kdc[3681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:13:27.959115 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.943137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.3.19.65:45839)
0925 09:13:27.943356 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:27.943363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:27.943948 (+ 585us) server_negotiation.cc:407] Connection header received
0925 09:13:27.944067 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:27.944071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:27.944201 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:27.944293 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:27.945866 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.946721 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.948008 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.948509 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.948626 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.953076 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:27.953089 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:27.953104 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:27.953137 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:27.956707 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:27.957338 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:27.957345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:27.957347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:27.957508 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:27.957838 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:27.957841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:27.957843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:27.958181 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:27.958397 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:27.958700 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 09:13:27.958864 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 09:13:27.966267 4186 tablet.cc:2369] T e032c2cd70f54ec0afa71ea2c5e3eb04 P a07f14cdf85a49f7ab01fb0c1b57001d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:13:27.974751 4052 tablet.cc:2369] T e032c2cd70f54ec0afa71ea2c5e3eb04 P 44a0e9c0a8874ae9b5e793b06335c7df: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 09:13:27 dist-test-slave-l6qh krb5kdc[3681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 09:13:27 dist-test-slave-l6qh krb5kdc[3681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:13:28.004148 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:27.990090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.3.19.65:45839)
0925 09:13:27.990349 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:27.990358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:27.990667 (+ 309us) server_negotiation.cc:407] Connection header received
0925 09:13:27.990885 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:27.990890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:27.990953 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:27.991056 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:27.992318 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.993061 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.994404 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:27.994813 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:27.994946 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:27.998952 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:27.998958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:27.998961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:27.998993 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:28.001577 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:28.002155 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:28.002168 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:28.002170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:28.002284 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:28.002675 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:28.002681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:28.002685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:28.002943 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:28.003241 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:28.003502 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 09:13:28.003746 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 09:13:28.006934 3917 tablet.cc:2369] T e032c2cd70f54ec0afa71ea2c5e3eb04 P aae3229168d745ba9e2dd6acf3948cd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 09:13:28 dist-test-slave-l6qh krb5kdc[3681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 09:13:28 dist-test-slave-l6qh krb5kdc[3681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:13:28.048146 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:28.034213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.3.19.65:45839)
0925 09:13:28.034456 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:28.034465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:28.034796 (+ 331us) server_negotiation.cc:407] Connection header received
0925 09:13:28.035004 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:28.035009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:28.035083 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:28.035192 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:28.036506 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:28.037316 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:28.038509 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:28.038910 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:28.039052 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:28.043079 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:28.043084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:28.043087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:28.043120 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:28.045898 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:28.046584 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:28.046588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:28.046590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:28.046713 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:28.047060 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:28.047063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:28.047065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:28.047344 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:28.047531 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:28.047726 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 09:13:28.047890 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 09:13:28.049974 3831 tablet_service.cc:2908] Rejecting scan request for tablet e032c2cd70f54ec0afa71ea2c5e3eb04: Uninitialized: clean time has not yet been initialized
I20260925 09:13:28.053351 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 3788
I20260925 09:13:28.065380 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:28.054191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.3.19.65:45839)
0925 09:13:28.054403 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:28.054409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:28.054631 (+ 222us) server_negotiation.cc:407] Connection header received
0925 09:13:28.054746 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:28.054749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:28.054808 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:28.054892 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:28.055954 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:28.056683 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:28.057884 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:28.058721 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:28.058829 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:28.060796 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:28.060800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:28.060803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:28.060832 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:28.063196 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:28.063758 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:28.063762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:28.063764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:28.063876 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:28.064152 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:28.064155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:28.064157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:28.064386 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:28.064551 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:28.064782 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 09:13:28.064947 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 09:13:28.065958 3911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:28.157133 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 3788
I20260925 09:13:28.174746 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 3922
Sep 25 09:13:28 dist-test-slave-l6qh krb5kdc[3681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:13:28.191588 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:28.175683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.3.19.66:43735)
0925 09:13:28.175875 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:28.175882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:28.175909 (+ 27us) server_negotiation.cc:407] Connection header received
0925 09:13:28.175973 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:28.175979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:28.176069 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:28.176177 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:28.177500 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:28.178255 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:28.179576 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:28.180013 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:28.180156 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:28.184816 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:28.184831 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:28.184847 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:28.184887 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:28.188477 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:28.189193 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:28.189202 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:28.189206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:28.189386 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:28.189806 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:28.189810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:28.189813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:28.190272 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:28.190597 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:28.191022 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 09:13:28.191240 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 09:13:28.192433 4046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:28.280609 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 3922
I20260925 09:13:28.297405 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 4056
Sep 25 09:13:28 dist-test-slave-l6qh krb5kdc[3681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:13:28.313015 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:28.298262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.3.19.67:41355)
0925 09:13:28.298478 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:28.298485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:28.298506 (+ 21us) server_negotiation.cc:407] Connection header received
0925 09:13:28.298643 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:28.298655 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:28.298720 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:28.298832 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:28.300063 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:28.300795 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:28.302666 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:28.303113 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:28.303252 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:28.307348 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:28.307363 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:28.307380 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:28.307418 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:28.310346 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:28.311095 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:28.311106 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:28.311109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:28.311285 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:28.311638 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:28.311641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:28.311643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:28.312005 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:28.312253 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:28.312456 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 09:13:28.312624 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 09:13:28.313673 4180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:28.401327 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 4056
I20260925 09:13:28.418305 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 3694
I20260925 09:13:28.419795 3757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:28.532122 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 3694
2026-09-25T09:13:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4024 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:13:28 dist-test-slave-l6qh krb5kdc[4233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:13:28 dist-test-slave-l6qh krb5kdc[4233](info): set up 2 sockets
Sep 25 09:13:28 dist-test-slave-l6qh krb5kdc[4233](info): commencing operation
krb5kdc: starting...
W20260925 09:13:29.622380 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:13:29 dist-test-slave-l6qh krb5kdc[4233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:13:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:13:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:13:29.833956 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:38853
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:38853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:30.163340 4246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:30.163739 4246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:30.163899 4246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:30.171929 4246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:13:30.172088 4246 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 09:13:30.172158 4246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:30.172254 4246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:30.172343 4246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:13:30.172427 4246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:13:30.182960 4246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:38853
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:38853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:30.184355 4246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:30.185820 4246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:30.198321 4254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:30.198436 4252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:30.198460 4251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:30.199529 4246 server_base.cc:1034] running on GCE node
I20260925 09:13:30.200443 4246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:30.202061 4246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:30.203315 4246 hybrid_clock.cc:648] HybridClock initialized: now 1790327610203235 us; error 52 us; skew 500 ppm
I20260925 09:13:30.203711 4246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:30 dist-test-slave-l6qh krb5kdc[4233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327610, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:30.207094 4246 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:13:30.208782 4246 webserver.cc:460] Webserver started at http://127.3.19.126:42951/ using document root <none> and password file <none>
I20260925 09:13:30.209585 4246 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:30.209722 4246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:30.210043 4246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:30.213073 4246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "4d0877a3c9a84055b6d35c1e5f373566"
format_stamp: "Formatted at 2026-09-25 09:13:30 on dist-test-slave-l6qh"
server_key: "30e478eea75f459db0133a3bf7ed4d98"
server_key_iv: "7806e4687be1544c6d5f8b943869b4bd"
server_key_version: "encryptionkey@0"
I20260925 09:13:30.214020 4246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "4d0877a3c9a84055b6d35c1e5f373566"
format_stamp: "Formatted at 2026-09-25 09:13:30 on dist-test-slave-l6qh"
server_key: "30e478eea75f459db0133a3bf7ed4d98"
server_key_iv: "7806e4687be1544c6d5f8b943869b4bd"
server_key_version: "encryptionkey@0"
I20260925 09:13:30.219616 4246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 09:13:30.223623 4261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:30.225116 4246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:13:30.225385 4246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "4d0877a3c9a84055b6d35c1e5f373566"
format_stamp: "Formatted at 2026-09-25 09:13:30 on dist-test-slave-l6qh"
server_key: "30e478eea75f459db0133a3bf7ed4d98"
server_key_iv: "7806e4687be1544c6d5f8b943869b4bd"
server_key_version: "encryptionkey@0"
I20260925 09:13:30.225651 4246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:30.254051 4246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:30.258601 4246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:30.258889 4246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:30.281090 4246 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:38853
I20260925 09:13:30.281097 4323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:38853 every 8 connection(s)
I20260925 09:13:30.282665 4246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:13:30.286589 4324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:30.293030 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 4246
I20260925 09:13:30.293308 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:13:30.294134 3149 external_mini_cluster.cc:1428] Setting key 1ace52c48d756fb79a391011ddc767b2
I20260925 09:13:30.298575 4324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566: Bootstrap starting.
I20260925 09:13:30.301846 4324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:30.302913 4324 log.cc:824] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:30.305811 4324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566: No bootstrap required, opened a new log
Sep 25 09:13:30 dist-test-slave-l6qh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:30.313665 4324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d0877a3c9a84055b6d35c1e5f373566" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38853 } }
I20260925 09:13:30.314064 4324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:30.314195 4324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0877a3c9a84055b6d35c1e5f373566, State: Initialized, Role: FOLLOWER
I20260925 09:13:30.314925 4324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d0877a3c9a84055b6d35c1e5f373566" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38853 } }
I20260925 09:13:30.315227 4324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:13:30.315384 4324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:13:30.315583 4324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:30.317466 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:30.296380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34978 (local address 127.3.19.126:38853)
0925 09:13:30.297200 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:30.297213 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:30.297333 (+ 120us) server_negotiation.cc:407] Connection header received
0925 09:13:30.299149 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:30.299173 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:30.299586 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:30.299992 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:30.301513 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:30.302913 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:30.304575 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:30.305312 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:30.305495 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:30.309584 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:30.309592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:30.309608 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:30.309650 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:30.312847 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:30.313673 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:30.313680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:30.313686 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:30.313848 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:30.314317 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:30.314323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:30.314326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:30.315068 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:30.315394 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:30.315857 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 09:13:30.316321 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":366,"threads_started":1}
I20260925 09:13:30.318529 4324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d0877a3c9a84055b6d35c1e5f373566" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38853 } }
I20260925 09:13:30.319171 4324 leader_election.cc:302] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d0877a3c9a84055b6d35c1e5f373566; no voters:
I20260925 09:13:30.319761 4324 leader_election.cc:288] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:13:30.319960 4329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:30.320884 4329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [term 1 LEADER]: Becoming Leader. State: Replica: 4d0877a3c9a84055b6d35c1e5f373566, State: Running, Role: LEADER
I20260925 09:13:30.321542 4329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d0877a3c9a84055b6d35c1e5f373566" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38853 } }
I20260925 09:13:30.321983 4324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:13:30.324612 4331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d0877a3c9a84055b6d35c1e5f373566. Latest consensus state: current_term: 1 leader_uuid: "4d0877a3c9a84055b6d35c1e5f373566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d0877a3c9a84055b6d35c1e5f373566" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38853 } } }
I20260925 09:13:30.324585 4330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d0877a3c9a84055b6d35c1e5f373566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d0877a3c9a84055b6d35c1e5f373566" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38853 } } }
I20260925 09:13:30.325091 4330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:30.325093 4331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566 [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:30.325891 4335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:13:30.331635 4335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:13:30.348347 4335 catalog_manager.cc:1347] Generated new cluster ID: d77d0c584166452b82a50501b9ff7f98
I20260925 09:13:30.348505 4335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:13:30.372888 4335 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:13:30.374845 4335 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:13:30.382736 4335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566: Generated new TSK 0
I20260925 09:13:30.383585 4335 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:13:30.446048 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:38853
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:30.788715 4350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:30.789074 4350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:30.789296 4350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:30.796751 4350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:30.796921 4350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:30.797078 4350 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 09:13:30.797221 4350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:13:30.807340 4350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.19.126:38853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:30.808971 4350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:30.810649 4350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:30.824594 4356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:30.825414 4355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:30.825388 4350 server_base.cc:1034] running on GCE node
W20260925 09:13:30.825320 4358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:30.826262 4350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:30.827242 4350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:30.828512 4350 hybrid_clock.cc:648] HybridClock initialized: now 1790327610828451 us; error 38 us; skew 500 ppm
I20260925 09:13:30.828931 4350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:30 dist-test-slave-l6qh krb5kdc[4233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327610, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:30.832191 4350 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:13:30.833844 4350 webserver.cc:460] Webserver started at http://127.3.19.65:37235/ using document root <none> and password file <none>
I20260925 09:13:30.834582 4350 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:30.834714 4350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:30.835039 4350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:30.838071 4350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "0a201f06ae1c46d7b4075c8b50740721"
format_stamp: "Formatted at 2026-09-25 09:13:30 on dist-test-slave-l6qh"
server_key: "ce9a717aafd8e0428dc18a23f38fc562"
server_key_iv: "579a6c933c6891c333cfc922e71f74a9"
server_key_version: "encryptionkey@0"
I20260925 09:13:30.838886 4350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "0a201f06ae1c46d7b4075c8b50740721"
format_stamp: "Formatted at 2026-09-25 09:13:30 on dist-test-slave-l6qh"
server_key: "ce9a717aafd8e0428dc18a23f38fc562"
server_key_iv: "579a6c933c6891c333cfc922e71f74a9"
server_key_version: "encryptionkey@0"
I20260925 09:13:30.844509 4350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:13:30.848879 4365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:30.850404 4350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:13:30.850616 4350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "0a201f06ae1c46d7b4075c8b50740721"
format_stamp: "Formatted at 2026-09-25 09:13:30 on dist-test-slave-l6qh"
server_key: "ce9a717aafd8e0428dc18a23f38fc562"
server_key_iv: "579a6c933c6891c333cfc922e71f74a9"
server_key_version: "encryptionkey@0"
I20260925 09:13:30.850865 4350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:30.889827 4350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:30.894549 4350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:30.894985 4350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:30.897876 4350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:30.897992 4350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:30.898188 4350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:30.898285 4350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 09:13:30 dist-test-slave-l6qh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327610, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:30.926338 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:30.898892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:49767 (local address 127.3.19.126:38853)
0925 09:13:30.899411 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:30.899416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:30.900801 (+ 1385us) server_negotiation.cc:407] Connection header received
0925 09:13:30.902992 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:30.902998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:30.903094 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:30.903217 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:30.906417 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:30.907470 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:30.910946 (+ 3476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:30.911546 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:30.911704 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:30.916801 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:30.916810 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:30.916818 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:30.916893 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:30.920457 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:30.921300 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:30.921309 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:30.921314 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:30.921486 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:30.922202 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:30.922210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:30.922214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:30.922645 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:30.925179 (+ 2534us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:30.925427 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 09:13:30.925754 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":280,"threads_started":1}
I20260925 09:13:30.928699 4375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:30.899669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38853 (local address 127.3.19.65:49767)
0925 09:13:30.900617 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:30.900659 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:30.902656 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:30.903726 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:30.903746 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:30.904818 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:30.906106 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:30.906144 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:30.907673 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:30.907680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:30.910735 (+ 3055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:30.910758 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:30.911889 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:30.911898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:30.912151 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:30.913091 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:30.913119 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:30.916462 (+ 3343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:30.920629 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:30.920640 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:30.920660 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:30.921126 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:30.921658 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:30.921663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:30.921667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:30.922016 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:30.922834 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:30.922851 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:30.923549 (+ 698us) client_negotiation.cc:769] Sending connection context
0925 09:13:30.923972 (+ 423us) client_negotiation.cc:240] Negotiation successful
0925 09:13:30.924789 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":419,"threads_started":1}
I20260925 09:13:30.943403 4350 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:39569
I20260925 09:13:30.943434 4486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:39569 every 8 connection(s)
I20260925 09:13:30.944962 4350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:13:30.947362 4487 heartbeater.cc:347] Connected to a master server at 127.3.19.126:38853
I20260925 09:13:30.947609 4487 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:30.948177 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 4350
I20260925 09:13:30.948421 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:13:30.948452 4487 heartbeater.cc:511] Master 127.3.19.126:38853 requested a full tablet report, sending...
I20260925 09:13:30.949050 3149 external_mini_cluster.cc:1428] Setting key e4b05b5085f2ca68a7eba009d9a5ef48
I20260925 09:13:30.950896 4278 ts_manager.cc:194] Registered new tserver with Master: 0a201f06ae1c46d7b4075c8b50740721 (127.3.19.65:39569)
I20260925 09:13:30.952339 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:30.937304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:49467 (local address 127.3.19.126:38853)
0925 09:13:30.937746 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:30.937754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:30.937787 (+ 33us) server_negotiation.cc:407] Connection header received
0925 09:13:30.937851 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:30.937855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:30.937911 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:30.938016 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:30.939856 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:30.940999 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:30.942625 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:30.943340 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:30.943497 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:30.945690 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:30.945698 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:30.945707 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:30.945753 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:30.949351 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:30.950091 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:30.950096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:30.950100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:30.950283 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:30.950694 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:30.950699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:30.950702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:30.951113 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:30.951296 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:30.951732 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 09:13:30.951951 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323}
I20260925 09:13:30.952373 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:30.936831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38853 (local address 127.3.19.65:49467)
0925 09:13:30.937524 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:30.937553 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:30.937704 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:30.938038 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:30.938043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:30.938921 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:30.939612 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:30.939631 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:30.941140 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:30.941144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:30.942468 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:30.942480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:30.943557 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:30.943565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:30.943749 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:30.944481 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:30.944499 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:30.945449 (+ 950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:30.949481 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:30.949487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:30.949492 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:30.949952 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:30.950411 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:30.950416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:30.950419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:30.950562 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:30.951269 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:30.951274 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:30.951458 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 09:13:30.951609 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 09:13:30.951941 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":521,"threads_started":1}
I20260925 09:13:30.953009 4278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:49767
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:13:30.993476 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:38853
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:31.327766 4491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:31.328064 4491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:31.328163 4491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:31.335819 4491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:31.335949 4491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:31.336036 4491 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 09:13:31.336118 4491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:13:31.346419 4491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.19.126:38853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:31.347714 4491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:31.349061 4491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:31.363080 4497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:31.363168 4496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:31.363143 4499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:31.364379 4491 server_base.cc:1034] running on GCE node
I20260925 09:13:31.365001 4491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:31.365998 4491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:31.367216 4491 hybrid_clock.cc:648] HybridClock initialized: now 1790327611367150 us; error 41 us; skew 500 ppm
I20260925 09:13:31.367542 4491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:31 dist-test-slave-l6qh krb5kdc[4233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327611, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:31.375851 4491 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:13:31.377532 4491 webserver.cc:460] Webserver started at http://127.3.19.66:39437/ using document root <none> and password file <none>
I20260925 09:13:31.378262 4491 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:31.378401 4491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:31.378728 4491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:31.381747 4491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "fe536b6430c24aa7b7d83153bff532a3"
format_stamp: "Formatted at 2026-09-25 09:13:31 on dist-test-slave-l6qh"
server_key: "4a4f3db9b96f504b03ab4266b602b581"
server_key_iv: "f02ad857e703f4e8d3591fb21d5caa4d"
server_key_version: "encryptionkey@0"
I20260925 09:13:31.382541 4491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "fe536b6430c24aa7b7d83153bff532a3"
format_stamp: "Formatted at 2026-09-25 09:13:31 on dist-test-slave-l6qh"
server_key: "4a4f3db9b96f504b03ab4266b602b581"
server_key_iv: "f02ad857e703f4e8d3591fb21d5caa4d"
server_key_version: "encryptionkey@0"
I20260925 09:13:31.387810 4491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 09:13:31.391541 4506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:31.392944 4491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:13:31.393132 4491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "fe536b6430c24aa7b7d83153bff532a3"
format_stamp: "Formatted at 2026-09-25 09:13:31 on dist-test-slave-l6qh"
server_key: "4a4f3db9b96f504b03ab4266b602b581"
server_key_iv: "f02ad857e703f4e8d3591fb21d5caa4d"
server_key_version: "encryptionkey@0"
I20260925 09:13:31.393411 4491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:31.411707 4491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:31.415992 4491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:31.416270 4491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:31.419204 4491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:31.419323 4491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:31.419523 4491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:31.419595 4491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 09:13:31 dist-test-slave-l6qh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327611, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:31.445465 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:31.419626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:48519 (local address 127.3.19.126:38853)
0925 09:13:31.419793 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:31.419798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:31.421524 (+ 1726us) server_negotiation.cc:407] Connection header received
0925 09:13:31.423868 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:31.423874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:31.423970 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:31.424081 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:31.426430 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:31.427766 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:31.430525 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:31.431305 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:31.431460 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:31.437712 (+ 6252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:31.437722 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:31.437730 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:31.437790 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:31.441509 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:31.442545 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:31.442551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:31.442562 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:31.442679 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:31.443284 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:31.443289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:31.443292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:31.443692 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:31.443848 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:31.444829 (+ 981us) server_negotiation.cc:299] Negotiation successful
0925 09:13:31.445059 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 09:13:31.446687 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:31.420348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38853 (local address 127.3.19.66:48519)
0925 09:13:31.421364 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:31.421403 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:31.423585 (+ 2182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:31.424392 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:31.424406 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:31.425293 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:31.426197 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:31.426230 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:31.427968 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:31.427976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:31.430355 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:31.430367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:31.431657 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:31.431668 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:31.432161 (+ 493us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:31.433115 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:31.433158 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:31.437322 (+ 4164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:31.441698 (+ 4376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:31.441712 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:31.441736 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:31.442392 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:31.442807 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:31.442812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:31.442816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:31.443158 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:31.443814 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:31.443823 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:31.444401 (+ 578us) client_negotiation.cc:769] Sending connection context
0925 09:13:31.444720 (+ 319us) client_negotiation.cc:240] Negotiation successful
0925 09:13:31.445362 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":383,"threads_started":1}
I20260925 09:13:31.467157 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:31.453219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38853 (local address 127.3.19.66:60223)
0925 09:13:31.454157 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:31.454183 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:31.454351 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:31.454700 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:31.454705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:31.455260 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:31.455549 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:31.455558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:31.456652 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:31.456659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:31.458124 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:31.458136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:31.458868 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:31.458876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:31.459064 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:31.459799 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:31.459816 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:31.460523 (+ 707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:31.464096 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:31.464101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:31.464109 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:31.464765 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:31.465168 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:31.465172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:31.465176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:31.465389 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:31.466324 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:31.466327 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:31.466492 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 09:13:31.466597 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 09:13:31.466848 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":581,"threads_started":1}
I20260925 09:13:31.467581 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:31.452963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:60223 (local address 127.3.19.126:38853)
0925 09:13:31.453160 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:31.453166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:31.454306 (+ 1140us) server_negotiation.cc:407] Connection header received
0925 09:13:31.454495 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:31.454500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:31.454571 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:31.454676 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:31.455698 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:31.456524 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:31.458276 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:31.458673 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:31.458807 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:31.460714 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:31.460719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:31.460722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:31.460752 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:31.463950 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:31.464914 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:31.464919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:31.464924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:31.465052 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:31.465550 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:31.465558 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:31.465564 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:31.466172 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:31.466376 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:31.466768 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 09:13:31.467082 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 09:13:31.481707 4491 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:44983
I20260925 09:13:31.481736 4625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:44983 every 8 connection(s)
I20260925 09:13:31.483016 4491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:13:31.485478 4626 heartbeater.cc:347] Connected to a master server at 127.3.19.126:38853
I20260925 09:13:31.485744 4626 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:31.486472 4626 heartbeater.cc:511] Master 127.3.19.126:38853 requested a full tablet report, sending...
I20260925 09:13:31.487870 4278 ts_manager.cc:194] Registered new tserver with Master: fe536b6430c24aa7b7d83153bff532a3 (127.3.19.66:44983)
I20260925 09:13:31.488767 4278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:48519
I20260925 09:13:31.493172 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 4491
I20260925 09:13:31.493432 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:13:31.493860 3149 external_mini_cluster.cc:1428] Setting key 6065179393457a612981684c9c289fab
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:13:31.538558 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:38853
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:31.894709 4630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:31.895077 4630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:31.895279 4630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:31.902877 4630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:31.903059 4630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:31.903224 4630 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 09:13:31.903365 4630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:13:31.913324 4630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.19.126:38853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:31.914685 4630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:31.916148 4630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:31.930138 4635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:31.930506 4638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:31.930622 4636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:31.932404 4630 server_base.cc:1034] running on GCE node
I20260925 09:13:31.933147 4630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:31.934263 4630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:31.935564 4630 hybrid_clock.cc:648] HybridClock initialized: now 1790327611935473 us; error 64 us; skew 500 ppm
I20260925 09:13:31.935950 4630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:31 dist-test-slave-l6qh krb5kdc[4233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327611, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:31.939653 4630 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:13:31.941395 4630 webserver.cc:460] Webserver started at http://127.3.19.67:35357/ using document root <none> and password file <none>
I20260925 09:13:31.942196 4630 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:31.942353 4630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:31.942708 4630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:31.946194 4630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "202643ea6cc24b16ac6130c66b6b6141"
format_stamp: "Formatted at 2026-09-25 09:13:31 on dist-test-slave-l6qh"
server_key: "aa9a9944e4ba278d27bf676830b137db"
server_key_iv: "8b9ebc4fd81443bac4c3ac7abfe6ef54"
server_key_version: "encryptionkey@0"
I20260925 09:13:31.947141 4630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "202643ea6cc24b16ac6130c66b6b6141"
format_stamp: "Formatted at 2026-09-25 09:13:31 on dist-test-slave-l6qh"
server_key: "aa9a9944e4ba278d27bf676830b137db"
server_key_iv: "8b9ebc4fd81443bac4c3ac7abfe6ef54"
server_key_version: "encryptionkey@0"
I20260925 09:13:31.952915 4630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 09:13:31.956913 4487 heartbeater.cc:503] Master 127.3.19.126:38853 was elected leader, sending a full tablet report...
I20260925 09:13:31.957185 4645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:31.959161 4630 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 09:13:31.959368 4630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "202643ea6cc24b16ac6130c66b6b6141"
format_stamp: "Formatted at 2026-09-25 09:13:31 on dist-test-slave-l6qh"
server_key: "aa9a9944e4ba278d27bf676830b137db"
server_key_iv: "8b9ebc4fd81443bac4c3ac7abfe6ef54"
server_key_version: "encryptionkey@0"
I20260925 09:13:31.959633 4630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:31.991972 4630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:31.996883 4630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:31.997257 4630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:32.000039 4630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:32.000167 4630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:32.000334 4630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:32.000429 4630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 09:13:32 dist-test-slave-l6qh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327611, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:32.024323 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.000963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:57733 (local address 127.3.19.126:38853)
0925 09:13:32.003368 (+ 2405us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.003374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.003398 (+ 24us) server_negotiation.cc:407] Connection header received
0925 09:13:32.004932 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.004938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.005020 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.005151 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:32.007468 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.008583 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.011363 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.012043 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.012191 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.016699 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:32.016708 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:32.016715 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:32.016763 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:32.020446 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:32.021380 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:32.021385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:32.021391 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:32.021522 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:32.022217 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:32.022220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:32.022222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:32.022603 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:32.022787 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.023862 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.024026 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2268,"thread_start_us":2160,"threads_started":1}
I20260925 09:13:32.026736 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.001607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38853 (local address 127.3.19.67:57733)
0925 09:13:32.002528 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:32.002574 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:32.004627 (+ 2053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:32.005438 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:32.005447 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:32.006263 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:32.007252 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.007280 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.008782 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.008791 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.011142 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.011202 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.012292 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.012300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.012533 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.013309 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:32.013341 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:32.016357 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:32.020679 (+ 4322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:32.020691 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:32.020717 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:32.021197 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:32.021739 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:32.021748 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:32.021754 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:32.022081 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:32.022771 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:32.022784 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:32.023381 (+ 597us) client_negotiation.cc:769] Sending connection context
0925 09:13:32.023788 (+ 407us) client_negotiation.cc:240] Negotiation successful
0925 09:13:32.024654 (+ 866us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":415,"threads_started":1}
I20260925 09:13:32.053592 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.036726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38853 (local address 127.3.19.67:49477)
0925 09:13:32.037582 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:32.037607 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:32.037842 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:32.038400 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:32.038406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:32.039176 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:32.039527 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.039541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.041218 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.041277 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.043009 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.043031 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.044310 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.044317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.044470 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.045383 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:32.045404 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:32.046601 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:32.050342 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:32.050348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:32.050352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:32.051198 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:32.051642 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:32.051647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:32.051651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:32.051837 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:32.052649 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:32.052653 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:32.052817 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 09:13:32.052937 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 09:13:32.053182 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":405,"threads_started":1}
I20260925 09:13:32.053951 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.036627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:49477 (local address 127.3.19.126:38853)
0925 09:13:32.036811 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.036815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.037719 (+ 904us) server_negotiation.cc:407] Connection header received
0925 09:13:32.037993 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.038001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.038097 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.038199 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:32.039761 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.041057 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.043205 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.044055 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.044219 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.046853 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:32.046858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:32.046861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:32.046893 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:32.050152 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:32.051360 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:32.051367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:32.051373 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:32.051514 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:32.051999 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:32.052008 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:32.052013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:32.052501 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:32.052682 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.053110 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.053437 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 09:13:32.061411 4630 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:39743
I20260925 09:13:32.061479 4766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:39743 every 8 connection(s)
I20260925 09:13:32.062652 4630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:13:32.064996 4767 heartbeater.cc:347] Connected to a master server at 127.3.19.126:38853
I20260925 09:13:32.065253 4767 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:32.065970 4767 heartbeater.cc:511] Master 127.3.19.126:38853 requested a full tablet report, sending...
I20260925 09:13:32.067350 4278 ts_manager.cc:194] Registered new tserver with Master: 202643ea6cc24b16ac6130c66b6b6141 (127.3.19.67:39743)
I20260925 09:13:32.068174 4278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:57733
I20260925 09:13:32.068897 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 4630
I20260925 09:13:32.069092 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:13:32.069643 3149 external_mini_cluster.cc:1428] Setting key 80b0b36ece900da70d954d421a9b1df1
I20260925 09:13:32.072611 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 09:13:32 dist-test-slave-l6qh krb5kdc[4233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327612, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 09:13:32 dist-test-slave-l6qh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327612, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:32.120177 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.105611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.3.19.126:38853)
0925 09:13:32.105862 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.105869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.106053 (+ 184us) server_negotiation.cc:407] Connection header received
0925 09:13:32.106337 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.106349 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.106447 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.106580 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:32.108196 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.108953 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.110461 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.110913 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.111061 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.115102 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:32.115107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:32.115110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:32.115147 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:32.117589 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:32.118233 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:32.118243 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:32.118246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:32.118366 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:32.118752 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:32.118757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:32.118760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:32.119057 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:32.119323 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.119609 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.119829 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 09:13:32.124747 4278 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:13:32.132021 4278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:13:32.155400 4780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.144705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.3.19.65:39569)
0925 09:13:32.145357 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.145368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.146338 (+ 970us) server_negotiation.cc:407] Connection header received
0925 09:13:32.146714 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.146723 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.147037 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.147231 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:32.148190 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.150519 (+ 2329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.153063 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.154145 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.154300 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.154535 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.154910 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.155077 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":288,"threads_started":1}
I20260925 09:13:32.155721 4781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.144695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:39569 (local address 127.0.0.1:55920)
0925 09:13:32.145948 (+ 1253us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:32.145997 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:32.146465 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:32.147551 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:32.147561 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:32.147600 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:32.147994 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.148009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.150690 (+ 2681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.150696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.152867 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.152880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.154425 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.154431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.154684 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.154721 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 09:13:32.154866 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 09:13:32.155139 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":30,"thread_start_us":1186,"threads_started":1}
I20260925 09:13:32.158687 4782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.144734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:39743 (local address 127.0.0.1:58036)
0925 09:13:32.146176 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:32.146199 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:32.146465 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:32.150358 (+ 3893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:32.150363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:32.150387 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:32.150956 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.150969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.152990 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.152995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.155726 (+ 2731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.155740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.157707 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.157725 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.157970 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.157992 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 09:13:32.158097 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 09:13:32.158233 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1294,"mutex_wait_us":168,"thread_start_us":686,"threads_started":1}
I20260925 09:13:32.159036 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.145292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.3.19.67:39743)
0925 09:13:32.149884 (+ 4592us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.149892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.149921 (+ 29us) server_negotiation.cc:407] Connection header received
0925 09:13:32.149997 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.150003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.150171 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.150364 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:32.151158 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.152825 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.156039 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.157483 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.157635 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.157854 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.158138 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.158296 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4431,"thread_start_us":4246,"threads_started":1}
I20260925 09:13:32.159199 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.146770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.3.19.66:44983)
0925 09:13:32.148977 (+ 2207us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.149017 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.149779 (+ 762us) server_negotiation.cc:407] Connection header received
0925 09:13:32.149884 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.149892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.150172 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.150366 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:32.151413 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.152825 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.155859 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.157188 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.157355 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.157549 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.158743 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.158913 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1977,"thread_start_us":1504,"threads_started":1}
I20260925 09:13:32.159466 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.146316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:44983 (local address 127.0.0.1:42422)
0925 09:13:32.149514 (+ 3198us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:32.149535 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:32.149657 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:32.150593 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:32.150600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:32.150626 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:32.151218 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.151234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.152990 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.152995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.155644 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.155658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.157746 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.157765 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.158043 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.158077 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 09:13:32.158165 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 09:13:32.158293 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3071,"mutex_wait_us":124,"thread_start_us":2914,"threads_started":1}
I20260925 09:13:32.160032 4416 tablet_service.cc:1467] Processing CreateTablet for tablet 071141c384c6407897d39acd10a84042 (DEFAULT_TABLE table=test-table [id=2ed51e87778a4e58b8fb8ac981d2e3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:32.161723 4416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071141c384c6407897d39acd10a84042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:32.162326 4697 tablet_service.cc:1467] Processing CreateTablet for tablet 071141c384c6407897d39acd10a84042 (DEFAULT_TABLE table=test-table [id=2ed51e87778a4e58b8fb8ac981d2e3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:32.162750 4556 tablet_service.cc:1467] Processing CreateTablet for tablet 071141c384c6407897d39acd10a84042 (DEFAULT_TABLE table=test-table [id=2ed51e87778a4e58b8fb8ac981d2e3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:32.163779 4697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071141c384c6407897d39acd10a84042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:32.164006 4556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071141c384c6407897d39acd10a84042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:32.174331 4786 tablet_bootstrap.cc:492] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721: Bootstrap starting.
I20260925 09:13:32.178431 4788 tablet_bootstrap.cc:492] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141: Bootstrap starting.
I20260925 09:13:32.179750 4787 tablet_bootstrap.cc:492] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3: Bootstrap starting.
I20260925 09:13:32.179854 4786 tablet_bootstrap.cc:654] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:32.181316 4786 log.cc:824] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:32.183502 4787 tablet_bootstrap.cc:654] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:32.184574 4788 tablet_bootstrap.cc:654] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:32.185252 4787 log.cc:824] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:32.185401 4786 tablet_bootstrap.cc:492] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721: No bootstrap required, opened a new log
I20260925 09:13:32.185704 4786 ts_tablet_manager.cc:1397] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260925 09:13:32.185902 4788 log.cc:824] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:32.188990 4787 tablet_bootstrap.cc:492] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3: No bootstrap required, opened a new log
I20260925 09:13:32.189419 4787 ts_tablet_manager.cc:1397] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260925 09:13:32.191205 4788 tablet_bootstrap.cc:492] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141: No bootstrap required, opened a new log
I20260925 09:13:32.191501 4788 ts_tablet_manager.cc:1397] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260925 09:13:32.197104 4786 raft_consensus.cc:348] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } }
I20260925 09:13:32.197680 4786 raft_consensus.cc:374] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:32.197817 4786 raft_consensus.cc:729] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a201f06ae1c46d7b4075c8b50740721, State: Initialized, Role: FOLLOWER
I20260925 09:13:32.198493 4786 consensus_queue.cc:260] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } }
I20260925 09:13:32.200227 4788 raft_consensus.cc:348] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } }
I20260925 09:13:32.200593 4788 raft_consensus.cc:374] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:32.200706 4788 raft_consensus.cc:729] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202643ea6cc24b16ac6130c66b6b6141, State: Initialized, Role: FOLLOWER
I20260925 09:13:32.201457 4788 consensus_queue.cc:260] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } }
I20260925 09:13:32.201509 4787 raft_consensus.cc:348] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } }
I20260925 09:13:32.202066 4787 raft_consensus.cc:374] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:32.202198 4787 raft_consensus.cc:729] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe536b6430c24aa7b7d83153bff532a3, State: Initialized, Role: FOLLOWER
I20260925 09:13:32.202364 4786 ts_tablet_manager.cc:1428] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
I20260925 09:13:32.203356 4767 heartbeater.cc:503] Master 127.3.19.126:38853 was elected leader, sending a full tablet report...
I20260925 09:13:32.203871 4788 ts_tablet_manager.cc:1428] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260925 09:13:32.202994 4787 consensus_queue.cc:260] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } }
I20260925 09:13:32.206023 4626 heartbeater.cc:503] Master 127.3.19.126:38853 was elected leader, sending a full tablet report...
I20260925 09:13:32.206813 4787 ts_tablet_manager.cc:1428] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3: Time spent starting tablet: real 0.017s user 0.006s sys 0.009s
W20260925 09:13:32.235183 4627 tablet.cc:2369] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:13:32.276086 4793 raft_consensus.cc:482] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:13:32.276551 4793 raft_consensus.cc:504] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } }
I20260925 09:13:32.278393 4793 leader_election.cc:288] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a201f06ae1c46d7b4075c8b50740721 (127.3.19.65:39569), fe536b6430c24aa7b7d83153bff532a3 (127.3.19.66:44983)
I20260925 09:13:32.285645 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.279346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:57179 (local address 127.3.19.66:44983)
0925 09:13:32.279563 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.279569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.280064 (+ 495us) server_negotiation.cc:407] Connection header received
0925 09:13:32.280256 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.280260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.280315 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.280424 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:32.280975 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.281795 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.283929 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.284693 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.284846 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.284953 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.285245 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.285393 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 09:13:32.286010 4795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.279192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:44983 (local address 127.3.19.67:57179)
0925 09:13:32.279957 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:32.279971 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:32.280114 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:32.280456 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:32.280460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:32.280485 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:32.280814 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.280828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.281930 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.281935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.283780 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.283792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.284897 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.284903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.285066 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.285089 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 09:13:32.285191 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 09:13:32.285356 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":54,"thread_start_us":440,"threads_started":1}
I20260925 09:13:32.286548 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.278766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:39569 (local address 127.3.19.67:37417)
0925 09:13:32.279012 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:32.279035 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:32.279186 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:32.279736 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:32.279740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:32.279761 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:32.279933 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.279940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.281180 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.281187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.283861 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.283875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.285556 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.285562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.286040 (+ 478us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.286075 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 09:13:32.286164 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 09:13:32.286288 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":29}
I20260925 09:13:32.287066 4581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071141c384c6407897d39acd10a84042" candidate_uuid: "202643ea6cc24b16ac6130c66b6b6141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe536b6430c24aa7b7d83153bff532a3" is_pre_election: true
I20260925 09:13:32.287560 4581 raft_consensus.cc:2454] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 202643ea6cc24b16ac6130c66b6b6141 in term 0.
I20260925 09:13:32.288096 4780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.278904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:37417 (local address 127.3.19.65:39569)
0925 09:13:32.279154 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.279161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.279184 (+ 23us) server_negotiation.cc:407] Connection header received
0925 09:13:32.279372 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.279377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.279486 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.279573 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:32.280084 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.280998 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.284044 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.285218 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.285378 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.285475 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.286471 (+ 996us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.286610 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 09:13:32.288198 4648 leader_election.cc:302] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202643ea6cc24b16ac6130c66b6b6141, fe536b6430c24aa7b7d83153bff532a3; no voters:
I20260925 09:13:32.288604 4793 raft_consensus.cc:2781] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:13:32.288853 4793 raft_consensus.cc:482] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:13:32.288968 4793 raft_consensus.cc:3037] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:32.289017 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071141c384c6407897d39acd10a84042" candidate_uuid: "202643ea6cc24b16ac6130c66b6b6141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a201f06ae1c46d7b4075c8b50740721" is_pre_election: true
I20260925 09:13:32.289554 4436 raft_consensus.cc:2454] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 202643ea6cc24b16ac6130c66b6b6141 in term 0.
I20260925 09:13:32.291431 4793 raft_consensus.cc:504] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } }
I20260925 09:13:32.292012 4793 leader_election.cc:288] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [CANDIDATE]: Term 1 election: Requested vote from peers 0a201f06ae1c46d7b4075c8b50740721 (127.3.19.65:39569), fe536b6430c24aa7b7d83153bff532a3 (127.3.19.66:44983)
I20260925 09:13:32.292490 4581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071141c384c6407897d39acd10a84042" candidate_uuid: "202643ea6cc24b16ac6130c66b6b6141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe536b6430c24aa7b7d83153bff532a3"
I20260925 09:13:32.292474 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071141c384c6407897d39acd10a84042" candidate_uuid: "202643ea6cc24b16ac6130c66b6b6141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a201f06ae1c46d7b4075c8b50740721"
I20260925 09:13:32.292662 4436 raft_consensus.cc:3037] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:32.292689 4581 raft_consensus.cc:3037] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:32.294994 4581 raft_consensus.cc:2454] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 202643ea6cc24b16ac6130c66b6b6141 in term 1.
I20260925 09:13:32.295410 4648 leader_election.cc:302] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202643ea6cc24b16ac6130c66b6b6141, fe536b6430c24aa7b7d83153bff532a3; no voters:
I20260925 09:13:32.295706 4793 raft_consensus.cc:2781] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:32.295866 4436 raft_consensus.cc:2454] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 202643ea6cc24b16ac6130c66b6b6141 in term 1.
I20260925 09:13:32.301599 4793 raft_consensus.cc:686] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [term 1 LEADER]: Becoming Leader. State: Replica: 202643ea6cc24b16ac6130c66b6b6141, State: Running, Role: LEADER
I20260925 09:13:32.302151 4793 consensus_queue.cc:237] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } }
I20260925 09:13:32.306785 4276 catalog_manager.cc:5495] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 reported cstate change: term changed from 0 to 1, leader changed from <none> to 202643ea6cc24b16ac6130c66b6b6141 (127.3.19.67). New cstate: current_term: 1 leader_uuid: "202643ea6cc24b16ac6130c66b6b6141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } health_report { overall_health: UNKNOWN } } }
W20260925 09:13:32.314289 4768 tablet.cc:2369] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:13:32.356740 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.350243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.3.19.67:39743)
0925 09:13:32.350473 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.350481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.350611 (+ 130us) server_negotiation.cc:407] Connection header received
0925 09:13:32.351092 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.351097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.351194 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.351369 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:13:32.351772 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.352741 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.354351 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.354813 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.354969 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.355505 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:13:32.356052 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:13:32.356221 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.356359 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.356450 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 09:13:32.366269 4581 raft_consensus.cc:1261] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3 [term 1 FOLLOWER]: Refusing update from remote peer 202643ea6cc24b16ac6130c66b6b6141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:32.366263 4436 raft_consensus.cc:1261] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721 [term 1 FOLLOWER]: Refusing update from remote peer 202643ea6cc24b16ac6130c66b6b6141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:32.367270 4796 consensus_queue.cc:1035] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:32.367640 4793 consensus_queue.cc:1035] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:32.399538 4277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:13:32.399895 4277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:13:32.404304 4277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566: Sending DeleteTablet for 3 replicas of tablet 071141c384c6407897d39acd10a84042
I20260925 09:13:32.405751 4697 tablet_service.cc:1514] Processing DeleteTablet for tablet 071141c384c6407897d39acd10a84042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:32 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:58036
I20260925 09:13:32.405761 4416 tablet_service.cc:1514] Processing DeleteTablet for tablet 071141c384c6407897d39acd10a84042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:32 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:55920
I20260925 09:13:32.406826 4556 tablet_service.cc:1514] Processing DeleteTablet for tablet 071141c384c6407897d39acd10a84042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:32 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:42422
I20260925 09:13:32.407430 4277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:13:32.408293 4277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:13:32.418103 4415 tablet_service.cc:1467] Processing CreateTablet for tablet 8c4073351feb4543b52c331eb813021d (DEFAULT_TABLE table=test-table [id=993053afc2b34d158254de6fefe23b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:32.418622 4696 tablet_service.cc:1467] Processing CreateTablet for tablet 8c4073351feb4543b52c331eb813021d (DEFAULT_TABLE table=test-table [id=993053afc2b34d158254de6fefe23b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:32.418864 4415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c4073351feb4543b52c331eb813021d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:32.419401 4696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c4073351feb4543b52c331eb813021d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:32.421116 4809 tablet_replica.cc:330] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721: stopping tablet replica
I20260925 09:13:32.422401 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 8c4073351feb4543b52c331eb813021d (DEFAULT_TABLE table=test-table [id=993053afc2b34d158254de6fefe23b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:32.423044 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c4073351feb4543b52c331eb813021d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:32.423306 4809 raft_consensus.cc:2229] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:13:32.423833 4809 raft_consensus.cc:2258] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:32.430840 4787 tablet_bootstrap.cc:492] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3: Bootstrap starting.
I20260925 09:13:32.432194 4786 tablet_bootstrap.cc:492] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721: Bootstrap starting.
I20260925 09:13:32.432787 4787 tablet_bootstrap.cc:654] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:32.433568 4788 tablet_bootstrap.cc:492] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141: Bootstrap starting.
I20260925 09:13:32.434281 4786 tablet_bootstrap.cc:654] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:32.435201 4809 ts_tablet_manager.cc:1905] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:32.435698 4788 tablet_bootstrap.cc:654] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:32.441668 4810 tablet_replica.cc:330] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3: stopping tablet replica
I20260925 09:13:32.442391 4810 raft_consensus.cc:2229] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:13:32.444604 4810 raft_consensus.cc:2258] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:32.446182 4808 tablet_replica.cc:330] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141: stopping tablet replica
I20260925 09:13:32.447335 4786 tablet_bootstrap.cc:492] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721: No bootstrap required, opened a new log
I20260925 09:13:32.447513 4786 ts_tablet_manager.cc:1397] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.003s
I20260925 09:13:32.447757 4810 ts_tablet_manager.cc:1905] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:32.448762 4808 raft_consensus.cc:2229] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:13:32.448491 4786 raft_consensus.cc:348] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.449479 4786 raft_consensus.cc:374] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:32.449582 4786 raft_consensus.cc:729] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a201f06ae1c46d7b4075c8b50740721, State: Initialized, Role: FOLLOWER
I20260925 09:13:32.449671 4808 raft_consensus.cc:2258] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:32.449892 4786 consensus_queue.cc:260] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.450477 4786 ts_tablet_manager.cc:1428] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 09:13:32.449147 4809 ts_tablet_manager.cc:1918] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:13:32.450851 4809 log.cc:1196] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/071141c384c6407897d39acd10a84042
I20260925 09:13:32.451421 4809 ts_tablet_manager.cc:1939] T 071141c384c6407897d39acd10a84042 P 0a201f06ae1c46d7b4075c8b50740721: Deleting consensus metadata
I20260925 09:13:32.451550 4788 tablet_bootstrap.cc:492] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141: No bootstrap required, opened a new log
I20260925 09:13:32.451697 4788 ts_tablet_manager.cc:1397] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.006s
I20260925 09:13:32.452488 4788 raft_consensus.cc:348] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.452644 4808 ts_tablet_manager.cc:1905] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:32.452708 4788 raft_consensus.cc:374] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:32.452801 4788 raft_consensus.cc:729] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202643ea6cc24b16ac6130c66b6b6141, State: Initialized, Role: FOLLOWER
I20260925 09:13:32.453260 4264 catalog_manager.cc:4841] TS 0a201f06ae1c46d7b4075c8b50740721 (127.3.19.65:39569): tablet 071141c384c6407897d39acd10a84042 (table test-table [id=2ed51e87778a4e58b8fb8ac981d2e3a7]) successfully deleted
I20260925 09:13:32.454521 4788 consensus_queue.cc:260] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.455111 4788 ts_tablet_manager.cc:1428] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 09:13:32.459443 4810 ts_tablet_manager.cc:1918] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:13:32.459631 4810 log.cc:1196] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/071141c384c6407897d39acd10a84042
I20260925 09:13:32.460104 4810 ts_tablet_manager.cc:1939] T 071141c384c6407897d39acd10a84042 P fe536b6430c24aa7b7d83153bff532a3: Deleting consensus metadata
I20260925 09:13:32.461663 4264 catalog_manager.cc:4841] TS fe536b6430c24aa7b7d83153bff532a3 (127.3.19.66:44983): tablet 071141c384c6407897d39acd10a84042 (table test-table [id=2ed51e87778a4e58b8fb8ac981d2e3a7]) successfully deleted
I20260925 09:13:32.462133 4787 tablet_bootstrap.cc:492] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3: No bootstrap required, opened a new log
I20260925 09:13:32.462307 4787 ts_tablet_manager.cc:1397] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.002s
I20260925 09:13:32.463023 4787 raft_consensus.cc:348] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.463248 4787 raft_consensus.cc:374] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:32.463425 4787 raft_consensus.cc:729] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe536b6430c24aa7b7d83153bff532a3, State: Initialized, Role: FOLLOWER
I20260925 09:13:32.463716 4787 consensus_queue.cc:260] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.464447 4787 ts_tablet_manager.cc:1428] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 09:13:32.464740 4808 ts_tablet_manager.cc:1918] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:13:32.464926 4808 log.cc:1196] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/071141c384c6407897d39acd10a84042
I20260925 09:13:32.465504 4808 ts_tablet_manager.cc:1939] T 071141c384c6407897d39acd10a84042 P 202643ea6cc24b16ac6130c66b6b6141: Deleting consensus metadata
I20260925 09:13:32.467705 4262 catalog_manager.cc:4841] TS 202643ea6cc24b16ac6130c66b6b6141 (127.3.19.67:39743): tablet 071141c384c6407897d39acd10a84042 (table test-table [id=2ed51e87778a4e58b8fb8ac981d2e3a7]) successfully deleted
I20260925 09:13:32.471939 4792 raft_consensus.cc:482] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:13:32.472157 4792 raft_consensus.cc:504] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.473868 4792 leader_election.cc:288] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe536b6430c24aa7b7d83153bff532a3 (127.3.19.66:44983), 202643ea6cc24b16ac6130c66b6b6141 (127.3.19.67:39743)
I20260925 09:13:32.480808 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.474495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:53227 (local address 127.3.19.66:44983)
0925 09:13:32.474677 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.474684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.475269 (+ 585us) server_negotiation.cc:407] Connection header received
0925 09:13:32.475509 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.475514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.475577 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.475664 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:32.476169 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.476909 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.479066 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.479834 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.479984 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.480085 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.480394 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.480516 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 09:13:32.480927 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.474092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:44983 (local address 127.3.19.65:53227)
0925 09:13:32.475150 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:32.475170 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:32.475372 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:32.475690 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:32.475694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:32.475725 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:32.476019 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.476028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.477048 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.477053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.478889 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.478900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.480049 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.480055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.480224 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.480246 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 09:13:32.480371 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 09:13:32.480512 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":1,"thread_start_us":671,"threads_started":1}
I20260925 09:13:32.481706 4581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4073351feb4543b52c331eb813021d" candidate_uuid: "0a201f06ae1c46d7b4075c8b50740721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe536b6430c24aa7b7d83153bff532a3" is_pre_election: true
I20260925 09:13:32.481993 4581 raft_consensus.cc:2454] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a201f06ae1c46d7b4075c8b50740721 in term 0.
I20260925 09:13:32.482622 4368 leader_election.cc:302] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a201f06ae1c46d7b4075c8b50740721, fe536b6430c24aa7b7d83153bff532a3; no voters:
I20260925 09:13:32.483047 4792 raft_consensus.cc:2781] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:13:32.483170 4792 raft_consensus.cc:482] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:13:32.483249 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.474620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:40769 (local address 127.3.19.67:39743)
0925 09:13:32.474793 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.474800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.475501 (+ 701us) server_negotiation.cc:407] Connection header received
0925 09:13:32.475779 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.475801 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.475874 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.475936 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:32.476731 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.478047 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.480854 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.482033 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.482162 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.482261 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.482836 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.482969 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 09:13:32.483258 4792 raft_consensus.cc:3037] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:32.483985 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.474138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:39743 (local address 127.3.19.65:40769)
0925 09:13:32.475403 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:32.475426 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:32.475597 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:32.476160 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:32.476164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:32.476189 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:32.476545 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.476556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.478223 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.478228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.480642 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.480658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.482452 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.482458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.482639 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.482667 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 09:13:32.483430 (+ 763us) client_negotiation.cc:240] Negotiation successful
0925 09:13:32.483594 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"mutex_wait_us":1,"thread_start_us":1001,"threads_started":1}
I20260925 09:13:32.485054 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4073351feb4543b52c331eb813021d" candidate_uuid: "0a201f06ae1c46d7b4075c8b50740721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202643ea6cc24b16ac6130c66b6b6141" is_pre_election: true
I20260925 09:13:32.485844 4722 raft_consensus.cc:2454] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a201f06ae1c46d7b4075c8b50740721 in term 0.
I20260925 09:13:32.486815 4792 raft_consensus.cc:504] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.487987 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4073351feb4543b52c331eb813021d" candidate_uuid: "0a201f06ae1c46d7b4075c8b50740721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202643ea6cc24b16ac6130c66b6b6141"
I20260925 09:13:32.488040 4581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4073351feb4543b52c331eb813021d" candidate_uuid: "0a201f06ae1c46d7b4075c8b50740721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe536b6430c24aa7b7d83153bff532a3"
I20260925 09:13:32.488212 4722 raft_consensus.cc:3037] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:32.488231 4581 raft_consensus.cc:3037] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:32.491571 4581 raft_consensus.cc:2454] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a201f06ae1c46d7b4075c8b50740721 in term 1.
I20260925 09:13:32.491571 4722 raft_consensus.cc:2454] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a201f06ae1c46d7b4075c8b50740721 in term 1.
I20260925 09:13:32.492007 4368 leader_election.cc:302] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a201f06ae1c46d7b4075c8b50740721, fe536b6430c24aa7b7d83153bff532a3; no voters:
I20260925 09:13:32.492656 4792 leader_election.cc:288] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [CANDIDATE]: Term 1 election: Requested vote from peers fe536b6430c24aa7b7d83153bff532a3 (127.3.19.66:44983), 202643ea6cc24b16ac6130c66b6b6141 (127.3.19.67:39743)
I20260925 09:13:32.492677 4816 raft_consensus.cc:2781] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:32.492964 4816 raft_consensus.cc:686] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [term 1 LEADER]: Becoming Leader. State: Replica: 0a201f06ae1c46d7b4075c8b50740721, State: Running, Role: LEADER
I20260925 09:13:32.493492 4816 consensus_queue.cc:237] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.498982 4277 catalog_manager.cc:5495] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a201f06ae1c46d7b4075c8b50740721 (127.3.19.65). New cstate: current_term: 1 leader_uuid: "0a201f06ae1c46d7b4075c8b50740721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } health_report { overall_health: UNKNOWN } } }
Sep 25 09:13:32 dist-test-slave-l6qh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327610, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:32.564648 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.547956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38853 (local address 127.0.0.1:34998)
0925 09:13:32.549180 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:32.549198 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:32.549389 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:32.549820 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:32.549824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:32.550547 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:32.550799 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.550807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.552291 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.552295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.553340 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.553351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.554393 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.554398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.554570 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.555334 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:32.555349 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:32.557812 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:32.562068 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:32.562077 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:32.562083 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:32.562506 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:32.562888 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:32.562891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:32.562893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:32.563066 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:32.563694 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:32.563735 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:32.563918 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 09:13:32.564059 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 09:13:32.564265 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":882,"threads_started":1}
I20260925 09:13:32.564756 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.547956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.3.19.126:38853)
0925 09:13:32.548162 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.548168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.549337 (+ 1169us) server_negotiation.cc:407] Connection header received
0925 09:13:32.549516 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.549522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.549600 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.549720 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:32.550954 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.552146 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.553492 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.554172 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.554340 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.558008 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:32.558015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:32.558020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:32.558056 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:32.561930 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:32.562635 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:32.562641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:32.562645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:32.562786 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:32.563175 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:32.563180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:32.563184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:32.563552 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:32.563694 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.564191 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.564426 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 09:13:32.570384 4277 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:34998:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 09:13:32.571406 4277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:13:32.578454 4415 tablet_service.cc:1467] Processing CreateTablet for tablet 434b4330490b4fe8906018cd09c17ef6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0fb00fe872f47d9b1578f6f713b3c8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 09:13:32.578608 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 434b4330490b4fe8906018cd09c17ef6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0fb00fe872f47d9b1578f6f713b3c8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 09:13:32.579066 4415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434b4330490b4fe8906018cd09c17ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:32.578902 4697 tablet_service.cc:1467] Processing CreateTablet for tablet 434b4330490b4fe8906018cd09c17ef6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0fb00fe872f47d9b1578f6f713b3c8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 09:13:32.579257 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434b4330490b4fe8906018cd09c17ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:32.579439 4697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434b4330490b4fe8906018cd09c17ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:32.587112 4786 tablet_bootstrap.cc:492] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: Bootstrap starting.
I20260925 09:13:32.587124 4788 tablet_bootstrap.cc:492] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141: Bootstrap starting.
I20260925 09:13:32.587327 4787 tablet_bootstrap.cc:492] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3: Bootstrap starting.
I20260925 09:13:32.589171 4787 tablet_bootstrap.cc:654] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:32.589310 4788 tablet_bootstrap.cc:654] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:32.589401 4786 tablet_bootstrap.cc:654] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:32.599869 4786 tablet_bootstrap.cc:492] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: No bootstrap required, opened a new log
I20260925 09:13:32.600065 4786 ts_tablet_manager.cc:1397] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260925 09:13:32.600207 4787 tablet_bootstrap.cc:492] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3: No bootstrap required, opened a new log
I20260925 09:13:32.600245 4788 tablet_bootstrap.cc:492] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141: No bootstrap required, opened a new log
I20260925 09:13:32.600417 4788 ts_tablet_manager.cc:1397] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260925 09:13:32.600448 4787 ts_tablet_manager.cc:1397] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260925 09:13:32.601325 4787 raft_consensus.cc:348] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.601666 4787 raft_consensus.cc:374] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:32.601529 4786 raft_consensus.cc:348] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.601833 4787 raft_consensus.cc:729] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe536b6430c24aa7b7d83153bff532a3, State: Initialized, Role: FOLLOWER
I20260925 09:13:32.601854 4786 raft_consensus.cc:374] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:32.601925 4786 raft_consensus.cc:729] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a201f06ae1c46d7b4075c8b50740721, State: Initialized, Role: FOLLOWER
I20260925 09:13:32.601807 4788 raft_consensus.cc:348] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.602062 4788 raft_consensus.cc:374] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:32.602147 4788 raft_consensus.cc:729] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202643ea6cc24b16ac6130c66b6b6141, State: Initialized, Role: FOLLOWER
I20260925 09:13:32.602201 4786 consensus_queue.cc:260] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.602325 4787 consensus_queue.cc:260] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.602420 4788 consensus_queue.cc:260] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.602687 4816 tablet_replica.cc:439] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } }
I20260925 09:13:32.602983 4816 tablet_replica.cc:442] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:32.602895 4796 tablet_replica.cc:439] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } }
I20260925 09:13:32.603157 4796 tablet_replica.cc:442] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:32.602994 4794 tablet_replica.cc:439] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } }
I20260925 09:13:32.603386 4794 tablet_replica.cc:442] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:32.604146 4788 ts_tablet_manager.cc:1428] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 09:13:32.605062 4786 ts_tablet_manager.cc:1428] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 09:13:32.605195 4787 ts_tablet_manager.cc:1428] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
W20260925 09:13:32.697953 4488 tablet.cc:2369] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:13:32.812227 4794 raft_consensus.cc:482] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:13:32.812481 4794 raft_consensus.cc:504] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.814154 4794 leader_election.cc:288] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a201f06ae1c46d7b4075c8b50740721 (127.3.19.65:39569), 202643ea6cc24b16ac6130c66b6b6141 (127.3.19.67:39743)
I20260925 09:13:32.827113 4816 raft_consensus.cc:482] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:13:32.827376 4816 raft_consensus.cc:504] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.828171 4816 leader_election.cc:288] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe536b6430c24aa7b7d83153bff532a3 (127.3.19.66:44983), 202643ea6cc24b16ac6130c66b6b6141 (127.3.19.67:39743)
I20260925 09:13:32.829159 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434b4330490b4fe8906018cd09c17ef6" candidate_uuid: "0a201f06ae1c46d7b4075c8b50740721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202643ea6cc24b16ac6130c66b6b6141" is_pre_election: true
I20260925 09:13:32.829489 4722 raft_consensus.cc:2454] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a201f06ae1c46d7b4075c8b50740721 in term 0.
I20260925 09:13:32.830092 4366 leader_election.cc:302] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a201f06ae1c46d7b4075c8b50740721, 202643ea6cc24b16ac6130c66b6b6141; no voters:
I20260925 09:13:32.830458 4816 raft_consensus.cc:2781] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:13:32.830560 4816 raft_consensus.cc:482] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:13:32.830677 4816 raft_consensus.cc:3037] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:32.832413 4581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434b4330490b4fe8906018cd09c17ef6" candidate_uuid: "0a201f06ae1c46d7b4075c8b50740721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe536b6430c24aa7b7d83153bff532a3" is_pre_election: true
I20260925 09:13:32.832664 4581 raft_consensus.cc:2454] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a201f06ae1c46d7b4075c8b50740721 in term 0.
I20260925 09:13:32.833851 4816 raft_consensus.cc:504] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.834901 4581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434b4330490b4fe8906018cd09c17ef6" candidate_uuid: "0a201f06ae1c46d7b4075c8b50740721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe536b6430c24aa7b7d83153bff532a3"
I20260925 09:13:32.835151 4581 raft_consensus.cc:3037] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:32.835098 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434b4330490b4fe8906018cd09c17ef6" candidate_uuid: "0a201f06ae1c46d7b4075c8b50740721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202643ea6cc24b16ac6130c66b6b6141"
I20260925 09:13:32.835348 4722 raft_consensus.cc:3037] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:32.838430 4581 raft_consensus.cc:2454] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a201f06ae1c46d7b4075c8b50740721 in term 1.
I20260925 09:13:32.838743 4722 raft_consensus.cc:2454] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a201f06ae1c46d7b4075c8b50740721 in term 1.
I20260925 09:13:32.839033 4368 leader_election.cc:302] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a201f06ae1c46d7b4075c8b50740721, fe536b6430c24aa7b7d83153bff532a3; no voters:
I20260925 09:13:32.839391 4792 raft_consensus.cc:2781] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:32.839876 4792 raft_consensus.cc:686] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [term 1 LEADER]: Becoming Leader. State: Replica: 0a201f06ae1c46d7b4075c8b50740721, State: Running, Role: LEADER
I20260925 09:13:32.840194 4792 consensus_queue.cc:237] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } }
I20260925 09:13:32.839370 4816 leader_election.cc:288] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [CANDIDATE]: Term 1 election: Requested vote from peers fe536b6430c24aa7b7d83153bff532a3 (127.3.19.66:44983), 202643ea6cc24b16ac6130c66b6b6141 (127.3.19.67:39743)
I20260925 09:13:32.843127 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.814408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:39569 (local address 127.3.19.66:42977)
0925 09:13:32.827607 (+ 13199us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:32.827634 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:32.827833 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:32.828886 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:32.828893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:32.828964 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:32.829674 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.829691 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.831817 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.831823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.837026 (+ 5203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.837041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.839133 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.839140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.839322 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.839369 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 09:13:32.839454 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 09:13:32.839601 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13046,"spinlock_wait_cycles":1408,"thread_start_us":14430,"threads_started":1}
I20260925 09:13:32.844218 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.814878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:58811 (local address 127.3.19.67:39743)
0925 09:13:32.815130 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.815140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.828794 (+ 13654us) server_negotiation.cc:407] Connection header received
0925 09:13:32.829306 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.829312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.829388 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.829461 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:32.830411 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.831477 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.840181 (+ 8704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.842098 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.842231 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.842332 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.843556 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.843793 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 09:13:32.845130 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.814588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:39743 (local address 127.3.19.66:58811)
0925 09:13:32.828708 (+ 14120us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:32.828727 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:32.829046 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:32.829667 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:32.829671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:32.829696 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:32.830181 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.830193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.831654 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.831660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.834958 (+ 3298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.834973 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.842601 (+ 7628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.842607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.842887 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.842942 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 09:13:32.844430 (+ 1488us) client_negotiation.cc:240] Negotiation successful
0925 09:13:32.844652 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13990,"thread_start_us":12764,"threads_started":1}
I20260925 09:13:32.845993 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.814566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:42977 (local address 127.3.19.65:39569)
0925 09:13:32.828141 (+ 13575us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.828153 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.828181 (+ 28us) server_negotiation.cc:407] Connection header received
0925 09:13:32.828459 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.828466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.828551 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.828674 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:32.829929 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.831630 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.837262 (+ 5632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.838680 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.838833 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.838976 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.845396 (+ 6420us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.845625 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13452,"thread_start_us":13757,"threads_started":1}
I20260925 09:13:32.846020 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434b4330490b4fe8906018cd09c17ef6" candidate_uuid: "fe536b6430c24aa7b7d83153bff532a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202643ea6cc24b16ac6130c66b6b6141" is_pre_election: true
I20260925 09:13:32.846643 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434b4330490b4fe8906018cd09c17ef6" candidate_uuid: "fe536b6430c24aa7b7d83153bff532a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a201f06ae1c46d7b4075c8b50740721" is_pre_election: true
I20260925 09:13:32.847007 4722 raft_consensus.cc:2379] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe536b6430c24aa7b7d83153bff532a3 in current term 1: Already voted for candidate 0a201f06ae1c46d7b4075c8b50740721 in this term.
I20260925 09:13:32.847600 4830 tablet_replica.cc:439] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: TxnStatusTablet state changed. Reason: New leader 0a201f06ae1c46d7b4075c8b50740721. Latest consensus state: current_term: 1 leader_uuid: "0a201f06ae1c46d7b4075c8b50740721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } }
I20260925 09:13:32.848026 4830 tablet_replica.cc:442] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: This TxnStatusTablet replica's current role is: LEADER
I20260925 09:13:32.848444 4507 leader_election.cc:302] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe536b6430c24aa7b7d83153bff532a3; no voters: 0a201f06ae1c46d7b4075c8b50740721, 202643ea6cc24b16ac6130c66b6b6141
I20260925 09:13:32.848619 4276 catalog_manager.cc:5495] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a201f06ae1c46d7b4075c8b50740721 (127.3.19.65). New cstate: current_term: 1 leader_uuid: "0a201f06ae1c46d7b4075c8b50740721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } health_report { overall_health: UNKNOWN } } }
I20260925 09:13:32.849399 4794 raft_consensus.cc:2726] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 09:13:32.850795 4832 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 09:13:32.851089 4832 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 09:13:32.896613 4830 consensus_queue.cc:1035] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:32.904244 4830 consensus_queue.cc:1035] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:32.929948 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.920108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.3.19.65:39569)
0925 09:13:32.920330 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.920339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.920362 (+ 23us) server_negotiation.cc:407] Connection header received
0925 09:13:32.920469 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.920477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.920607 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.920765 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:13:32.921565 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.922945 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.925401 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.926215 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.926362 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.927379 (+ 1017us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:13:32.928215 (+ 836us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:13:32.928376 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.929276 (+ 900us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.929668 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 09:13:32.929576 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.918865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:39569 (local address 127.0.0.1:55922)
0925 09:13:32.919465 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:32.919491 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:32.919670 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:32.921002 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:32.921009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:32.921028 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 09:13:32.921323 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.921338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.923179 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.923188 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.925183 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:32.925204 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.926827 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.926833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:32.927038 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.927130 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 09:13:32.928439 (+ 1309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 09:13:32.928476 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 09:13:32.928614 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 09:13:32.928839 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370}
I20260925 09:13:32.938271 4722 raft_consensus.cc:1261] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141 [term 1 FOLLOWER]: Refusing update from remote peer 0a201f06ae1c46d7b4075c8b50740721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:32.938740 4581 raft_consensus.cc:1261] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3 [term 1 FOLLOWER]: Refusing update from remote peer 0a201f06ae1c46d7b4075c8b50740721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:32.938896 4830 consensus_queue.cc:1035] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:32.939472 4830 consensus_queue.cc:1035] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:32.946182 4794 tablet_replica.cc:439] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3: TxnStatusTablet state changed. Reason: New leader 0a201f06ae1c46d7b4075c8b50740721. Latest consensus state: current_term: 1 leader_uuid: "0a201f06ae1c46d7b4075c8b50740721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } }
I20260925 09:13:32.946491 4794 tablet_replica.cc:442] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:32.948428 4835 mvcc.cc:205] Tried to move back new op lower bound from 7333181902584594432 to 7333181902196101120. Current Snapshot: MvccSnapshot[applied={T|T < 7333181902584594432}]
I20260925 09:13:32.955400 4794 tablet_replica.cc:439] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a201f06ae1c46d7b4075c8b50740721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } }
I20260925 09:13:32.955734 4794 tablet_replica.cc:442] T 434b4330490b4fe8906018cd09c17ef6 P fe536b6430c24aa7b7d83153bff532a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:32.961964 4836 mvcc.cc:205] Tried to move back new op lower bound from 7333181902584594432 to 7333181902196101120. Current Snapshot: MvccSnapshot[applied={T|T < 7333181902584594432 or (T in {7333181902584594432})}]
I20260925 09:13:32.970949 4796 tablet_replica.cc:439] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141: TxnStatusTablet state changed. Reason: New leader 0a201f06ae1c46d7b4075c8b50740721. Latest consensus state: current_term: 1 leader_uuid: "0a201f06ae1c46d7b4075c8b50740721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } }
I20260925 09:13:32.971395 4796 tablet_replica.cc:442] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:32.975997 4839 mvcc.cc:205] Tried to move back new op lower bound from 7333181902584594432 to 7333181902196101120. Current Snapshot: MvccSnapshot[applied={T|T < 7333181902584594432}]
I20260925 09:13:32.983155 4830 tablet_replica.cc:439] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a201f06ae1c46d7b4075c8b50740721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } }
I20260925 09:13:32.983450 4830 tablet_replica.cc:442] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: This TxnStatusTablet replica's current role is: LEADER
I20260925 09:13:32.984902 4830 tablet_replica.cc:439] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a201f06ae1c46d7b4075c8b50740721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } }
I20260925 09:13:32.985139 4830 tablet_replica.cc:442] T 434b4330490b4fe8906018cd09c17ef6 P 0a201f06ae1c46d7b4075c8b50740721: This TxnStatusTablet replica's current role is: LEADER
I20260925 09:13:32.986217 4796 tablet_replica.cc:439] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a201f06ae1c46d7b4075c8b50740721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a201f06ae1c46d7b4075c8b50740721" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39569 } } peers { permanent_uuid: "fe536b6430c24aa7b7d83153bff532a3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 44983 } } peers { permanent_uuid: "202643ea6cc24b16ac6130c66b6b6141" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39743 } } }
I20260925 09:13:32.986560 4796 tablet_replica.cc:442] T 434b4330490b4fe8906018cd09c17ef6 P 202643ea6cc24b16ac6130c66b6b6141: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:32.995095 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:32.989388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.3.19.65:39569)
0925 09:13:32.989563 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:32.989568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:32.989956 (+ 388us) server_negotiation.cc:407] Connection header received
0925 09:13:32.990138 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:32.990145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:32.990301 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:32.990463 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:13:32.991025 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.991858 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.993369 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:32.993807 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:32.993960 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:32.994366 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:13:32.994475 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:13:32.994582 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:32.994713 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 09:13:32.994791 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 09:13:33.011825 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:33.006263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:35115 (local address 127.3.19.65:39569)
0925 09:13:33.006419 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:33.006424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:33.007000 (+ 576us) server_negotiation.cc:407] Connection header received
0925 09:13:33.007203 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:33.007208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:33.007279 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:33.007378 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:13:33.007813 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.008570 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.010081 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.010528 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.010689 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:33.011081 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:13:33.011185 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:13:33.011279 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:33.011408 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 09:13:33.011479 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 09:13:33.011924 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:33.006307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:39569 (local address 127.3.19.65:35115)
0925 09:13:33.006898 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:33.006913 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:33.007083 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:33.007389 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:33.007393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:33.007403 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 09:13:33.007673 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:33.007681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.008713 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.008720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:33.009936 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:33.009951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.010703 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.010709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:33.010850 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:33.010906 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 09:13:33.011278 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 09:13:33.011288 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 09:13:33.011393 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 09:13:33.011525 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":1,"thread_start_us":314,"threads_started":1}
I20260925 09:13:33.087558 4278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:13:33.087817 4278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:13:33.090920 4278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d0877a3c9a84055b6d35c1e5f373566: Sending DeleteTablet for 3 replicas of tablet 8c4073351feb4543b52c331eb813021d
I20260925 09:13:33.091825 4415 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c4073351feb4543b52c331eb813021d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:33 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:55920
I20260925 09:13:33.091853 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c4073351feb4543b52c331eb813021d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:33 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:42422
I20260925 09:13:33.092275 4697 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c4073351feb4543b52c331eb813021d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:33 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:58036
I20260925 09:13:33.103708 4854 tablet_replica.cc:330] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3: stopping tablet replica
I20260925 09:13:33.104550 4854 raft_consensus.cc:2229] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:13:33.105171 4854 raft_consensus.cc:2258] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:33.107277 4854 ts_tablet_manager.cc:1905] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:33.110888 4856 tablet_replica.cc:330] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141: stopping tablet replica
I20260925 09:13:33.111138 4856 raft_consensus.cc:2229] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:13:33.111428 4856 raft_consensus.cc:2258] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:33.113047 4856 ts_tablet_manager.cc:1905] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:33.115268 4855 tablet_replica.cc:330] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721: stopping tablet replica
I20260925 09:13:33.115463 4855 raft_consensus.cc:2229] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:13:33.116184 4855 raft_consensus.cc:2258] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:33.118780 4854 ts_tablet_manager.cc:1918] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:13:33.119659 4855 ts_tablet_manager.cc:1905] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:33.122301 4854 log.cc:1196] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/8c4073351feb4543b52c331eb813021d
I20260925 09:13:33.123055 4854 ts_tablet_manager.cc:1939] T 8c4073351feb4543b52c331eb813021d P fe536b6430c24aa7b7d83153bff532a3: Deleting consensus metadata
I20260925 09:13:33.124490 4264 catalog_manager.cc:4841] TS fe536b6430c24aa7b7d83153bff532a3 (127.3.19.66:44983): tablet 8c4073351feb4543b52c331eb813021d (table test-table [id=993053afc2b34d158254de6fefe23b4f]) successfully deleted
I20260925 09:13:33.127132 4856 ts_tablet_manager.cc:1918] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:13:33.127290 4856 log.cc:1196] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/8c4073351feb4543b52c331eb813021d
I20260925 09:13:33.127736 4856 ts_tablet_manager.cc:1939] T 8c4073351feb4543b52c331eb813021d P 202643ea6cc24b16ac6130c66b6b6141: Deleting consensus metadata
I20260925 09:13:33.128865 4262 catalog_manager.cc:4841] TS 202643ea6cc24b16ac6130c66b6b6141 (127.3.19.67:39743): tablet 8c4073351feb4543b52c331eb813021d (table test-table [id=993053afc2b34d158254de6fefe23b4f]) successfully deleted
I20260925 09:13:33.130972 4855 ts_tablet_manager.cc:1918] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:13:33.131188 4855 log.cc:1196] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/8c4073351feb4543b52c331eb813021d
I20260925 09:13:33.131827 4855 ts_tablet_manager.cc:1939] T 8c4073351feb4543b52c331eb813021d P 0a201f06ae1c46d7b4075c8b50740721: Deleting consensus metadata
I20260925 09:13:33.132958 4264 catalog_manager.cc:4841] TS 0a201f06ae1c46d7b4075c8b50740721 (127.3.19.65:39569): tablet 8c4073351feb4543b52c331eb813021d (table test-table [id=993053afc2b34d158254de6fefe23b4f]) successfully deleted
Sep 25 09:13:33 dist-test-slave-l6qh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327612, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:13:33.147650 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:33.113651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.3.19.65:39569)
0925 09:13:33.113825 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:33.113832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:33.124071 (+ 10239us) server_negotiation.cc:407] Connection header received
0925 09:13:33.128884 (+ 4813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:33.128891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:33.129019 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:33.129100 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:33.131607 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.133027 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.135466 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.136062 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.136194 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:33.141112 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:33.141126 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:33.141142 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:33.141181 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:33.144431 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:33.145319 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:33.145327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:33.145330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:33.145471 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:33.145889 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:33.145901 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:33.145904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:33.146383 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:33.146731 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:33.147006 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 09:13:33.147265 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 09:13:33.148471 4482 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55938
I20260925 09:13:33.168344 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:33.153016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.3.19.126:38853)
0925 09:13:33.153597 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:33.153604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:33.153624 (+ 20us) server_negotiation.cc:407] Connection header received
0925 09:13:33.153714 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:33.153721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:33.153793 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:33.153907 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:33.155421 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.156453 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.157760 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.158403 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.158530 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:33.160852 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:33.160857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:33.160860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:33.160894 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:33.164884 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:33.165836 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:33.165844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:33.165848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:33.166024 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:33.166517 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:33.166524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:33.166526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:33.166857 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:33.167136 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:33.167578 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 09:13:33.167858 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":323,"threads_started":1}
W20260925 09:13:33.169741 4278 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35012
I20260925 09:13:33.171835 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 4350
I20260925 09:13:33.185380 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:33.173050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55954 (local address 127.3.19.65:39569)
0925 09:13:33.173218 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:33.173268 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:33.173414 (+ 146us) server_negotiation.cc:407] Connection header received
0925 09:13:33.173602 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:33.173608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:33.173685 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:33.173785 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:33.174904 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.175638 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.176814 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.177362 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.177496 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:33.179747 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:33.179754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:33.179759 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:33.179801 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:33.182659 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:33.183404 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:33.183409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:33.183413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:33.183541 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:33.183907 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:33.183912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:33.183916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:33.184190 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:33.184471 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:33.184746 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 09:13:33.184970 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 09:13:33.186249 4482 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55954
W20260925 09:13:33.187011 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 4350: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:13:33.187252 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 4350
I20260925 09:13:33.206646 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 4491
Sep 25 09:13:33 dist-test-slave-l6qh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327612, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:13:33.222492 4863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:33.207435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.3.19.66:44983)
0925 09:13:33.207930 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:33.207935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:33.207951 (+ 16us) server_negotiation.cc:407] Connection header received
0925 09:13:33.208004 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:33.208007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:33.208070 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:33.208162 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:33.209365 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.210084 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.211357 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.211767 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.211867 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:33.216753 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:33.216767 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:33.216783 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:33.216819 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:33.219681 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:33.220485 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:33.220492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:33.220495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:33.220623 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:33.220988 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:33.220991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:33.220993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:33.221413 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:33.221633 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:33.221950 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 09:13:33.222173 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":250,"threads_started":1}
W20260925 09:13:33.223309 4621 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42434
W20260925 09:13:33.224092 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 4491: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:13:33.224244 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 4491
I20260925 09:13:33.243338 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 4630
Sep 25 09:13:33 dist-test-slave-l6qh krb5kdc[4233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327612, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:13:33.259068 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:33.244055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.3.19.67:39743)
0925 09:13:33.244224 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:33.244228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:33.244248 (+ 20us) server_negotiation.cc:407] Connection header received
0925 09:13:33.244419 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:33.244424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:33.244494 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:33.244586 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:33.245884 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.246731 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.247940 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:33.248447 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:33.248577 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:33.253113 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:33.253130 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:33.253147 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:33.253190 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:33.256186 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:33.256935 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:33.256942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:33.256945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:33.257091 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:33.257530 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:33.257537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:33.257540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:33.257888 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:33.258226 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:33.258475 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 09:13:33.258715 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 09:13:33.259756 4762 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58046
W20260925 09:13:33.260381 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 4630: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:13:33.260510 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 4630
I20260925 09:13:33.281636 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 4246
I20260925 09:13:33.282915 4319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:33.395473 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 4246
2026-09-25T09:13:33Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4862 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:13:33 dist-test-slave-l6qh krb5kdc[4867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:13:33 dist-test-slave-l6qh krb5kdc[4867](info): set up 2 sockets
Sep 25 09:13:33 dist-test-slave-l6qh krb5kdc[4867](info): commencing operation
krb5kdc: starting...
W20260925 09:13:34.510454 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:13:34 dist-test-slave-l6qh krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:13:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:13:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:13:34.712893 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:45681
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:45681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:35.037735 4880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:35.038057 4880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:35.038214 4880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:35.045678 4880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:13:35.045802 4880 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 09:13:35.045886 4880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:35.045982 4880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:35.046077 4880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:13:35.046190 4880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:13:35.056183 4880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:45681
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:45681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:35.057601 4880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:35.058997 4880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:35.071126 4885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:35.071126 4886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:35.071302 4888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:35.072031 4880 server_base.cc:1034] running on GCE node
I20260925 09:13:35.072911 4880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:35.074373 4880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:35.075598 4880 hybrid_clock.cc:648] HybridClock initialized: now 1790327615075541 us; error 28 us; skew 500 ppm
I20260925 09:13:35.075963 4880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:35 dist-test-slave-l6qh krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:35.079159 4880 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:13:35.080701 4880 webserver.cc:460] Webserver started at http://127.3.19.126:42851/ using document root <none> and password file <none>
I20260925 09:13:35.081509 4880 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:35.081638 4880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:35.081964 4880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:35.084995 4880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "b2298906d4cd4abb836d8f29b8013b0c"
format_stamp: "Formatted at 2026-09-25 09:13:35 on dist-test-slave-l6qh"
server_key: "421b7e98c9b0805c1f56fa92e3a6ff2f"
server_key_iv: "60b68699065ccd7d351d7e3a8bf392d0"
server_key_version: "encryptionkey@0"
I20260925 09:13:35.085867 4880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "b2298906d4cd4abb836d8f29b8013b0c"
format_stamp: "Formatted at 2026-09-25 09:13:35 on dist-test-slave-l6qh"
server_key: "421b7e98c9b0805c1f56fa92e3a6ff2f"
server_key_iv: "60b68699065ccd7d351d7e3a8bf392d0"
server_key_version: "encryptionkey@0"
I20260925 09:13:35.091325 4880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:13:35.094980 4895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:35.096344 4880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:13:35.096555 4880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "b2298906d4cd4abb836d8f29b8013b0c"
format_stamp: "Formatted at 2026-09-25 09:13:35 on dist-test-slave-l6qh"
server_key: "421b7e98c9b0805c1f56fa92e3a6ff2f"
server_key_iv: "60b68699065ccd7d351d7e3a8bf392d0"
server_key_version: "encryptionkey@0"
I20260925 09:13:35.096793 4880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:35.113211 4880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:35.117784 4880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:35.118073 4880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:35.140616 4880 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:45681
I20260925 09:13:35.140637 4956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:45681 every 8 connection(s)
I20260925 09:13:35.142138 4880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:13:35.146238 4957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:35.151849 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 4880
I20260925 09:13:35.152088 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:13:35.152748 3149 external_mini_cluster.cc:1428] Setting key 683154b2e39aaa76357cd0b8c98cd505
I20260925 09:13:35.157120 4957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c: Bootstrap starting.
I20260925 09:13:35.160979 4957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:35.162170 4957 log.cc:824] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:35.165530 4957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c: No bootstrap required, opened a new log
Sep 25 09:13:35 dist-test-slave-l6qh krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:35.175997 4957 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2298906d4cd4abb836d8f29b8013b0c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 45681 } }
I20260925 09:13:35.176470 4957 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:35.176575 4957 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2298906d4cd4abb836d8f29b8013b0c, State: Initialized, Role: FOLLOWER
I20260925 09:13:35.177322 4957 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2298906d4cd4abb836d8f29b8013b0c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 45681 } }
I20260925 09:13:35.177613 4957 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:13:35.177786 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:35.155005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.3.19.126:45681)
0925 09:13:35.155824 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:35.155836 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:35.155876 (+ 40us) server_negotiation.cc:407] Connection header received
0925 09:13:35.157642 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:35.157669 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:35.158073 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:35.158403 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:35.160318 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:35.161809 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:35.163396 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:35.164398 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:35.164590 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:35.168776 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:35.168787 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:35.168810 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:35.168860 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:35.173008 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:35.173853 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:35.173864 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:35.173879 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:35.174106 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:35.174454 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:35.174460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:35.174463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:35.175244 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:35.175528 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:35.175865 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 09:13:35.176317 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":397,"threads_started":1}
I20260925 09:13:35.177764 4957 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:13:35.177976 4957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:35.180430 4957 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2298906d4cd4abb836d8f29b8013b0c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 45681 } }
I20260925 09:13:35.181032 4957 leader_election.cc:302] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2298906d4cd4abb836d8f29b8013b0c; no voters:
I20260925 09:13:35.181696 4957 leader_election.cc:288] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:13:35.181844 4962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:35.182654 4962 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [term 1 LEADER]: Becoming Leader. State: Replica: b2298906d4cd4abb836d8f29b8013b0c, State: Running, Role: LEADER
I20260925 09:13:35.183231 4962 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2298906d4cd4abb836d8f29b8013b0c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 45681 } }
I20260925 09:13:35.183935 4957 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:13:35.185956 4963 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2298906d4cd4abb836d8f29b8013b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2298906d4cd4abb836d8f29b8013b0c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 45681 } } }
I20260925 09:13:35.186210 4964 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2298906d4cd4abb836d8f29b8013b0c. Latest consensus state: current_term: 1 leader_uuid: "b2298906d4cd4abb836d8f29b8013b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2298906d4cd4abb836d8f29b8013b0c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 45681 } } }
I20260925 09:13:35.186488 4964 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:35.186487 4963 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:35.187713 4970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:13:35.193763 4970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:13:35.204509 4970 catalog_manager.cc:1347] Generated new cluster ID: 125905bd015444ab816964a77e99f02a
I20260925 09:13:35.204644 4970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:13:35.235898 4970 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:13:35.237366 4970 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:13:35.244097 4970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c: Generated new TSK 0
I20260925 09:13:35.244972 4970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:13:35.291049 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:45681
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:35.625797 4983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:35.626091 4983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:35.626194 4983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:35.633831 4983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:35.633951 4983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:35.634040 4983 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 09:13:35.634093 4983 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 09:13:35.634155 4983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:13:35.644161 4983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.19.126:45681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:35.645480 4983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:35.646785 4983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:35.662999 4988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:35.662933 4991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:35.662961 4989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:35.664369 4983 server_base.cc:1034] running on GCE node
I20260925 09:13:35.664992 4983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:35.666082 4983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:35.667316 4983 hybrid_clock.cc:648] HybridClock initialized: now 1790327615667240 us; error 42 us; skew 500 ppm
I20260925 09:13:35.667693 4983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:35 dist-test-slave-l6qh krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:35.671134 4983 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:13:35.672693 4983 webserver.cc:460] Webserver started at http://127.3.19.65:39423/ using document root <none> and password file <none>
I20260925 09:13:35.673504 4983 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:35.673642 4983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:35.673974 4983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:35.677214 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0"
format_stamp: "Formatted at 2026-09-25 09:13:35 on dist-test-slave-l6qh"
server_key: "ab8ec57a80c2534fd439fea43be522e9"
server_key_iv: "347d4d6fbede4a22682adce87b8051f2"
server_key_version: "encryptionkey@0"
I20260925 09:13:35.678158 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0"
format_stamp: "Formatted at 2026-09-25 09:13:35 on dist-test-slave-l6qh"
server_key: "ab8ec57a80c2534fd439fea43be522e9"
server_key_iv: "347d4d6fbede4a22682adce87b8051f2"
server_key_version: "encryptionkey@0"
I20260925 09:13:35.684001 4983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 09:13:35.688896 4998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:35.690588 4983 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 09:13:35.690788 4983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0"
format_stamp: "Formatted at 2026-09-25 09:13:35 on dist-test-slave-l6qh"
server_key: "ab8ec57a80c2534fd439fea43be522e9"
server_key_iv: "347d4d6fbede4a22682adce87b8051f2"
server_key_version: "encryptionkey@0"
I20260925 09:13:35.690999 4983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:35.710582 4983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:35.715279 4983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:35.715597 4983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:35.718560 4983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:35.718664 4983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:35.718854 4983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:35.718973 4983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 09:13:35 dist-test-slave-l6qh krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:35.746120 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:35.719064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:58397 (local address 127.3.19.126:45681)
0925 09:13:35.719935 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:35.719942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:35.721082 (+ 1140us) server_negotiation.cc:407] Connection header received
0925 09:13:35.724360 (+ 3278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:35.724369 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:35.724474 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:35.724589 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:35.728057 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:35.729055 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:35.732523 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:35.733387 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:35.733565 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:35.738714 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:35.738723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:35.738732 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:35.738790 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:35.741797 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:35.742983 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:35.742991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:35.742996 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:35.743172 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:35.743787 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:35.743795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:35.743801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:35.744303 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:35.744516 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:35.745304 (+ 788us) server_negotiation.cc:299] Negotiation successful
0925 09:13:35.745590 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":644,"threads_started":1}
I20260925 09:13:35.747079 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:35.719860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:45681 (local address 127.3.19.65:58397)
0925 09:13:35.720850 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:35.720920 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:35.723911 (+ 2991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:35.725311 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:35.725329 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:35.726444 (+ 1115us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:35.727784 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:35.727834 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:35.729318 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:35.729323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:35.732244 (+ 2921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:35.732278 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:35.733652 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:35.733658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:35.733892 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:35.734815 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:35.734839 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:35.738387 (+ 3548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:35.742018 (+ 3631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:35.742027 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:35.742043 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:35.742777 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:35.743292 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:35.743297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:35.743301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:35.743602 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:35.744487 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:35.744494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:35.744875 (+ 381us) client_negotiation.cc:769] Sending connection context
0925 09:13:35.745111 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 09:13:35.745795 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":400,"threads_started":1}
I20260925 09:13:35.769891 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:35.754029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:47537 (local address 127.3.19.126:45681)
0925 09:13:35.754291 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:35.754301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:35.754805 (+ 504us) server_negotiation.cc:407] Connection header received
0925 09:13:35.754973 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:35.754982 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:35.755100 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:35.755248 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:35.756144 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:35.757613 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:35.759701 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:35.760268 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:35.760425 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:35.763372 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:35.763381 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:35.763388 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:35.763433 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:35.766839 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:35.767908 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:35.767914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:35.767918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:35.768052 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:35.768413 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:35.768419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:35.768423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:35.768690 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:35.768844 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:35.769296 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 09:13:35.769537 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 09:13:35.770061 4983 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:45725
I20260925 09:13:35.770140 5086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:35.754037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:45681 (local address 127.3.19.65:47537)
0925 09:13:35.754696 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:35.754722 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:35.754852 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:35.755261 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:35.755264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:35.755701 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:35.756001 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:35.756010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:35.757797 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:35.757806 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:35.759474 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:35.759496 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:35.760490 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:35.760496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:35.760700 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:35.761464 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:35.761493 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:35.763136 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:35.767058 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:35.767067 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:35.767071 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:35.767752 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:35.768163 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:35.768168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:35.768172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:35.768305 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:35.768814 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:35.768819 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:35.769012 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 09:13:35.769175 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 09:13:35.769594 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":367,"threads_started":1}
I20260925 09:13:35.770447 5119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:45725 every 8 connection(s)
I20260925 09:13:35.771680 4983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:13:35.773923 5120 heartbeater.cc:347] Connected to a master server at 127.3.19.126:45681
I20260925 09:13:35.774181 5120 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:35.774886 5120 heartbeater.cc:511] Master 127.3.19.126:45681 requested a full tablet report, sending...
I20260925 09:13:35.776737 4912 ts_manager.cc:194] Registered new tserver with Master: 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65:45725)
I20260925 09:13:35.778283 4912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:58397
I20260925 09:13:35.780905 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 4983
I20260925 09:13:35.781049 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:13:35.781687 3149 external_mini_cluster.cc:1428] Setting key 81a4ef50aae87965fe13d48e11cf08c3
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:13:35.824992 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:45681
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:36.157013 5124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:36.157420 5124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:36.157606 5124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:36.164978 5124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:36.165146 5124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:36.165345 5124 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 09:13:36.165437 5124 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 09:13:36.165555 5124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:13:36.175370 5124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.19.126:45681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:36.176745 5124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:36.178113 5124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:36.191641 5132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:36.191761 5129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:36.191730 5130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:36.192472 5124 server_base.cc:1034] running on GCE node
I20260925 09:13:36.193271 5124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:36.194314 5124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:36.195504 5124 hybrid_clock.cc:648] HybridClock initialized: now 1790327616195439 us; error 60 us; skew 500 ppm
I20260925 09:13:36.195914 5124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:36 dist-test-slave-l6qh krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:36.199148 5124 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:13:36.200691 5124 webserver.cc:460] Webserver started at http://127.3.19.66:39461/ using document root <none> and password file <none>
I20260925 09:13:36.201472 5124 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:36.201603 5124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:36.201921 5124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:36.204887 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "24e3ca0e87a44506a1d8cf084e98cc90"
format_stamp: "Formatted at 2026-09-25 09:13:36 on dist-test-slave-l6qh"
server_key: "62e050a54385bab54476430f90e56308"
server_key_iv: "346ad7f947d0bcb0e88cb5fd8024ca84"
server_key_version: "encryptionkey@0"
I20260925 09:13:36.205679 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "24e3ca0e87a44506a1d8cf084e98cc90"
format_stamp: "Formatted at 2026-09-25 09:13:36 on dist-test-slave-l6qh"
server_key: "62e050a54385bab54476430f90e56308"
server_key_iv: "346ad7f947d0bcb0e88cb5fd8024ca84"
server_key_version: "encryptionkey@0"
I20260925 09:13:36.210841 5124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 09:13:36.214366 5139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:36.215744 5124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 09:13:36.215909 5124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "24e3ca0e87a44506a1d8cf084e98cc90"
format_stamp: "Formatted at 2026-09-25 09:13:36 on dist-test-slave-l6qh"
server_key: "62e050a54385bab54476430f90e56308"
server_key_iv: "346ad7f947d0bcb0e88cb5fd8024ca84"
server_key_version: "encryptionkey@0"
I20260925 09:13:36.216120 5124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:36.234422 5124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:36.238616 5124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:36.238917 5124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:36.241626 5124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:36.241717 5124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:36.241886 5124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:36.241971 5124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 09:13:36 dist-test-slave-l6qh krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:36.268244 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.242420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:57677 (local address 127.3.19.126:45681)
0925 09:13:36.242623 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:36.242629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:36.244044 (+ 1415us) server_negotiation.cc:407] Connection header received
0925 09:13:36.246300 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:36.246306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:36.246398 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:36.246525 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:36.248777 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.249919 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.252653 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.253162 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.253355 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.260791 (+ 7436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:36.260796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:36.260800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:36.260836 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:36.264559 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:36.265813 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:36.265821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:36.265825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:36.265953 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:36.266430 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:36.266435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:36.266439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:36.266731 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:36.266881 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:36.267665 (+ 784us) server_negotiation.cc:299] Negotiation successful
0925 09:13:36.267890 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 09:13:36.269481 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.242947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:45681 (local address 127.3.19.66:57677)
0925 09:13:36.243848 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:36.243902 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:36.246001 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:36.247008 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:36.247016 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:36.247790 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:36.248588 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.248612 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.250083 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.250090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.252479 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.252491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.253606 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.253613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.254123 (+ 510us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.255034 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:36.255075 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:36.260468 (+ 5393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:36.264786 (+ 4318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:36.264795 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:36.264813 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:36.265659 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:36.266075 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:36.266080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:36.266084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:36.266303 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:36.266871 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:36.266880 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:36.267278 (+ 398us) client_negotiation.cc:769] Sending connection context
0925 09:13:36.267555 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 09:13:36.268134 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":419,"threads_started":1}
I20260925 09:13:36.295835 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.276763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:45681 (local address 127.3.19.66:45607)
0925 09:13:36.277600 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:36.277624 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:36.277825 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:36.278251 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:36.278259 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:36.279170 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:36.279896 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.279915 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.281115 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.281123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.283483 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.283504 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.284443 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.284452 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.284696 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.285541 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:36.285573 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:36.287058 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:36.292138 (+ 5080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:36.292143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:36.292146 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:36.292971 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:36.293592 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:36.293597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:36.293600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:36.293781 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:36.294696 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:36.294702 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:36.294901 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 09:13:36.295108 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 09:13:36.295456 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":384,"threads_started":1}
I20260925 09:13:36.296093 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.276811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:45607 (local address 127.3.19.126:45681)
0925 09:13:36.277126 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:36.277131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:36.277754 (+ 623us) server_negotiation.cc:407] Connection header received
0925 09:13:36.277989 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:36.277997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:36.278081 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:36.278180 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:36.280130 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.280936 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.283690 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.284225 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.284352 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.287312 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:36.287318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:36.287321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:36.287359 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:36.291910 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:36.293152 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:36.293160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:36.293165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:36.293434 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:36.293980 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:36.293986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:36.293989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:36.294543 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:36.294727 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:36.295271 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 09:13:36.295592 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260925 09:13:36.300659 5124 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:38795
I20260925 09:13:36.300701 5259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:38795 every 8 connection(s)
I20260925 09:13:36.302109 5124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:13:36.304265 5260 heartbeater.cc:347] Connected to a master server at 127.3.19.126:45681
I20260925 09:13:36.304505 5260 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:36.304561 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 5124
I20260925 09:13:36.304728 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:13:36.305150 3149 external_mini_cluster.cc:1428] Setting key 48ca7a8f69af909f6e5c6925bacf4922
I20260925 09:13:36.305281 5260 heartbeater.cc:511] Master 127.3.19.126:45681 requested a full tablet report, sending...
I20260925 09:13:36.306788 4911 ts_manager.cc:194] Registered new tserver with Master: 24e3ca0e87a44506a1d8cf084e98cc90 (127.3.19.66:38795)
I20260925 09:13:36.307965 4911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:57677
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:13:36.348407 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:45681
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:36.681035 5264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:36.681409 5264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:36.681609 5264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:36.689347 5264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:36.689460 5264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:36.689634 5264 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 09:13:36.689716 5264 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 09:13:36.689847 5264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:13:36.700304 5264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.19.126:45681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:36.701817 5264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:36.703187 5264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:36.719484 5269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:36.719455 5270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:36.719970 5272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:36.720523 5264 server_base.cc:1034] running on GCE node
I20260925 09:13:36.721319 5264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:36.722318 5264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:36.723556 5264 hybrid_clock.cc:648] HybridClock initialized: now 1790327616723511 us; error 41 us; skew 500 ppm
I20260925 09:13:36.723899 5264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:36 dist-test-slave-l6qh krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:36.732290 5264 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:13:36.734027 5264 webserver.cc:460] Webserver started at http://127.3.19.67:37923/ using document root <none> and password file <none>
I20260925 09:13:36.734790 5264 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:36.734937 5264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:36.735256 5264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:36.738317 5264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "0b326626ea524046967381848bde308b"
format_stamp: "Formatted at 2026-09-25 09:13:36 on dist-test-slave-l6qh"
server_key: "c0dcaa05806d2304ae4579365d1ed047"
server_key_iv: "0e6bafc0f4cd4981ca1d14962e34f0cd"
server_key_version: "encryptionkey@0"
I20260925 09:13:36.739143 5264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "0b326626ea524046967381848bde308b"
format_stamp: "Formatted at 2026-09-25 09:13:36 on dist-test-slave-l6qh"
server_key: "c0dcaa05806d2304ae4579365d1ed047"
server_key_iv: "0e6bafc0f4cd4981ca1d14962e34f0cd"
server_key_version: "encryptionkey@0"
I20260925 09:13:36.744499 5264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:13:36.748304 5279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:36.750022 5264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:13:36.750218 5264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "0b326626ea524046967381848bde308b"
format_stamp: "Formatted at 2026-09-25 09:13:36 on dist-test-slave-l6qh"
server_key: "c0dcaa05806d2304ae4579365d1ed047"
server_key_iv: "0e6bafc0f4cd4981ca1d14962e34f0cd"
server_key_version: "encryptionkey@0"
I20260925 09:13:36.750463 5264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:36.779695 5264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:36.780961 5120 heartbeater.cc:503] Master 127.3.19.126:45681 was elected leader, sending a full tablet report...
I20260925 09:13:36.784444 5264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:36.784780 5264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:36.787679 5264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:36.787815 5264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:36.787928 5264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:36.788014 5264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 09:13:36 dist-test-slave-l6qh krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:36.815739 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.788381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:35071 (local address 127.3.19.126:45681)
0925 09:13:36.788556 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:36.788560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:36.790424 (+ 1864us) server_negotiation.cc:407] Connection header received
0925 09:13:36.792538 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:36.792544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:36.792637 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:36.792764 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:36.794910 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.795933 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.799426 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.800252 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.800498 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.806711 (+ 6213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:36.806719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:36.806728 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:36.806792 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:36.810977 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:36.812377 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:36.812384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:36.812388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:36.812583 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:36.813321 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:36.813329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:36.813335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:36.813790 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:36.813974 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:36.815290 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0925 09:13:36.815474 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 09:13:36.818344 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.789370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:45681 (local address 127.3.19.67:35071)
0925 09:13:36.790252 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:36.790299 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:36.792205 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:36.793044 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:36.793056 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:36.793679 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:36.794665 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.794700 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.796124 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.796133 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.799172 (+ 3039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.799211 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.800655 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.800664 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.800993 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.801967 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:36.802009 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:36.806360 (+ 4351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:36.811183 (+ 4823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:36.811201 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:36.811227 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:36.812159 (+ 932us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:36.812742 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:36.812751 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:36.812759 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:36.813125 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:36.813969 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:36.813983 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:36.814744 (+ 761us) client_negotiation.cc:769] Sending connection context
0925 09:13:36.815162 (+ 418us) client_negotiation.cc:240] Negotiation successful
0925 09:13:36.816096 (+ 934us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":398,"threads_started":1}
I20260925 09:13:36.832784 5264 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:36073
I20260925 09:13:36.832898 5399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:36073 every 8 connection(s)
I20260925 09:13:36.834499 5264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:13:36.837901 5400 heartbeater.cc:347] Connected to a master server at 127.3.19.126:45681
I20260925 09:13:36.838047 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 5264
I20260925 09:13:36.838173 5400 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:36.838222 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:13:36.838814 3149 external_mini_cluster.cc:1428] Setting key eaf6802faa47092e846f531c7734fa6d
I20260925 09:13:36.839118 5400 heartbeater.cc:511] Master 127.3.19.126:45681 requested a full tablet report, sending...
I20260925 09:13:36.840761 4911 ts_manager.cc:194] Registered new tserver with Master: 0b326626ea524046967381848bde308b (127.3.19.67:36073)
I20260925 09:13:36.841558 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.824877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:46339 (local address 127.3.19.126:45681)
0925 09:13:36.825069 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:36.825074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:36.826306 (+ 1232us) server_negotiation.cc:407] Connection header received
0925 09:13:36.826664 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:36.826670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:36.826747 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:36.826810 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:36.828380 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.829316 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.831346 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.832153 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.832333 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.834493 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:36.834501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:36.834505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:36.834550 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:36.838467 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:36.839333 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:36.839339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:36.839343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:36.839490 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:36.839907 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:36.839912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:36.839915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:36.840297 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:36.840510 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:36.840954 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 09:13:36.841181 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 09:13:36.841663 5379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.825331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:45681 (local address 127.3.19.67:46339)
0925 09:13:36.826154 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:36.826177 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:36.826476 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:36.827040 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:36.827047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:36.827689 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:36.828161 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.828176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.829545 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.829552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.831191 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.831203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.832441 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.832450 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.832673 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.833434 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:36.833450 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:36.834249 (+ 799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:36.838618 (+ 4369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:36.838624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:36.838633 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:36.839202 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:36.839606 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:36.839612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:36.839615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:36.839746 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:36.840481 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:36.840487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:36.840706 (+ 219us) client_negotiation.cc:769] Sending connection context
0925 09:13:36.840868 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 09:13:36.841205 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":506,"threads_started":1}
I20260925 09:13:36.842015 4911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:35071
I20260925 09:13:36.842319 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 09:13:36 dist-test-slave-l6qh krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 09:13:36 dist-test-slave-l6qh krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:36.892335 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.877885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.3.19.126:45681)
0925 09:13:36.878119 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:36.878125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:36.878283 (+ 158us) server_negotiation.cc:407] Connection header received
0925 09:13:36.878465 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:36.878469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:36.878532 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:36.878620 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:36.880036 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.880802 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.882117 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.882532 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.882626 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.886720 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:36.886726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:36.886730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:36.886767 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:36.889685 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:36.890364 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:36.890369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:36.890371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:36.890511 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:36.890856 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:36.890859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:36.890861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:36.891139 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:36.891337 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:36.891639 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 09:13:36.891832 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 09:13:36.896922 4912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:13:36.904085 4912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:13:36.927727 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.916832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:38795 (local address 127.0.0.1:36490)
0925 09:13:36.919622 (+ 2790us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:36.919641 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:36.919841 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:36.920872 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:36.920878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:36.920909 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:36.921367 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.921381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.923164 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.923175 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.925408 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.925418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.926682 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.926688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.926898 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.926949 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 09:13:36.927102 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 09:13:36.927316 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2657,"thread_start_us":2513,"threads_started":1}
I20260925 09:13:36.928084 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.916606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:45725 (local address 127.0.0.1:51348)
0925 09:13:36.918153 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:36.918203 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:36.918373 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:36.919888 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:36.919896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:36.919942 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:36.920402 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.920415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.922613 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.922619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.925156 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.925170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.927125 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.927131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.927383 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.927427 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 09:13:36.927522 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 09:13:36.927641 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1299,"mutex_wait_us":40,"thread_start_us":1152,"threads_started":1}
I20260925 09:13:36.928890 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.917446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.3.19.66:38795)
0925 09:13:36.918839 (+ 1393us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:36.918846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:36.919724 (+ 878us) server_negotiation.cc:407] Connection header received
0925 09:13:36.920121 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:36.920129 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:36.920373 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:36.920563 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:36.921538 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.923023 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.925566 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.926447 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.926606 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.926847 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:36.927401 (+ 554us) server_negotiation.cc:299] Negotiation successful
0925 09:13:36.927582 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1261,"thread_start_us":1154,"threads_started":1}
I20260925 09:13:36.929081 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.917005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.3.19.65:45725)
0925 09:13:36.919180 (+ 2175us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:36.919186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:36.919256 (+ 70us) server_negotiation.cc:407] Connection header received
0925 09:13:36.919352 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:36.919358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:36.919636 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:36.920626 (+ 990us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:36.920834 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.922351 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.925351 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.926909 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.927050 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.927264 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:36.928567 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0925 09:13:36.928791 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2052,"thread_start_us":1960,"threads_started":1}
I20260925 09:13:36.930794 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.919722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:36073 (local address 127.0.0.1:60296)
0925 09:13:36.922620 (+ 2898us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:36.922639 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:36.922783 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:36.923724 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:36.923729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:36.923754 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:36.924337 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.924353 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.926105 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.926111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.928261 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:36.928275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.929978 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.929984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:36.930156 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.930188 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 09:13:36.930284 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 09:13:36.930413 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2763,"mutex_wait_us":28,"thread_start_us":2592,"threads_started":1}
I20260925 09:13:36.930794 5418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:36.920193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60296 (local address 127.3.19.67:36073)
0925 09:13:36.922112 (+ 1919us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:36.922125 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:36.922762 (+ 637us) server_negotiation.cc:407] Connection header received
0925 09:13:36.922935 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:36.922941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:36.923191 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:36.923391 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:36.924580 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.925936 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.928447 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:36.929778 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:36.929949 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:36.930176 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:36.930325 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 09:13:36.930481 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1718,"thread_start_us":1504,"threads_started":1}
I20260925 09:13:36.932940 5189 tablet_service.cc:1467] Processing CreateTablet for tablet 566a8002e0d1476fbf7005bb34da7f9d (DEFAULT_TABLE table=test-table [id=7dc3b6ecc1124fb4954e2a0256a28b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:36.932940 5050 tablet_service.cc:1467] Processing CreateTablet for tablet 566a8002e0d1476fbf7005bb34da7f9d (DEFAULT_TABLE table=test-table [id=7dc3b6ecc1124fb4954e2a0256a28b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:36.934495 5189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566a8002e0d1476fbf7005bb34da7f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:36.934474 5050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566a8002e0d1476fbf7005bb34da7f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:36.934725 5329 tablet_service.cc:1467] Processing CreateTablet for tablet 566a8002e0d1476fbf7005bb34da7f9d (DEFAULT_TABLE table=test-table [id=7dc3b6ecc1124fb4954e2a0256a28b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:36.935909 5329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566a8002e0d1476fbf7005bb34da7f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:36.948710 5419 tablet_bootstrap.cc:492] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b: Bootstrap starting.
I20260925 09:13:36.952231 5419 tablet_bootstrap.cc:654] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:36.953419 5419 log.cc:824] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:36.954136 5421 tablet_bootstrap.cc:492] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90: Bootstrap starting.
I20260925 09:13:36.955212 5420 tablet_bootstrap.cc:492] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Bootstrap starting.
I20260925 09:13:36.958359 5421 tablet_bootstrap.cc:654] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:36.958949 5420 tablet_bootstrap.cc:654] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:36.959261 5419 tablet_bootstrap.cc:492] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b: No bootstrap required, opened a new log
I20260925 09:13:36.959569 5419 ts_tablet_manager.cc:1397] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260925 09:13:36.959707 5421 log.cc:824] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:36.960130 5420 log.cc:824] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:36.963157 5421 tablet_bootstrap.cc:492] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90: No bootstrap required, opened a new log
I20260925 09:13:36.963348 5420 tablet_bootstrap.cc:492] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0: No bootstrap required, opened a new log
I20260925 09:13:36.963474 5421 ts_tablet_manager.cc:1397] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260925 09:13:36.963601 5420 ts_tablet_manager.cc:1397] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 09:13:36.968567 5419 raft_consensus.cc:348] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:36.969027 5419 raft_consensus.cc:374] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:36.969177 5419 raft_consensus.cc:729] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b326626ea524046967381848bde308b, State: Initialized, Role: FOLLOWER
I20260925 09:13:36.969975 5419 consensus_queue.cc:260] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:36.971233 5400 heartbeater.cc:503] Master 127.3.19.126:45681 was elected leader, sending a full tablet report...
I20260925 09:13:36.971995 5419 ts_tablet_manager.cc:1428] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b: Time spent starting tablet: real 0.012s user 0.007s sys 0.007s
I20260925 09:13:36.974819 5421 raft_consensus.cc:348] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:36.974819 5420 raft_consensus.cc:348] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:36.975286 5421 raft_consensus.cc:374] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:36.975284 5420 raft_consensus.cc:374] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:36.975387 5420 raft_consensus.cc:729] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d9fb7c7e27548b8a56c8a052dcf5dc0, State: Initialized, Role: FOLLOWER
I20260925 09:13:36.975438 5421 raft_consensus.cc:729] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e3ca0e87a44506a1d8cf084e98cc90, State: Initialized, Role: FOLLOWER
I20260925 09:13:36.976081 5420 consensus_queue.cc:260] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:36.976081 5421 consensus_queue.cc:260] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:36.977267 5260 heartbeater.cc:503] Master 127.3.19.126:45681 was elected leader, sending a full tablet report...
I20260925 09:13:36.978477 5420 ts_tablet_manager.cc:1428] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260925 09:13:36.979833 5421 ts_tablet_manager.cc:1428] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
W20260925 09:13:37.024206 5121 tablet.cc:2369] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:13:37.053687 5427 raft_consensus.cc:482] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 09:13:37.054172 5261 tablet.cc:2369] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:13:37.054122 5427 raft_consensus.cc:504] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.056037 5427 leader_election.cc:288] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65:45725), 0b326626ea524046967381848bde308b (127.3.19.67:36073)
I20260925 09:13:37.062950 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.056359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:44065 (local address 127.3.19.65:45725)
0925 09:13:37.056603 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:37.056613 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:37.057434 (+ 821us) server_negotiation.cc:407] Connection header received
0925 09:13:37.057613 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:37.057620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:37.057706 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:37.057826 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:37.058381 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.059105 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.061115 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.061958 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.062119 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.062235 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:37.062544 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 09:13:37.062675 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 09:13:37.062950 5418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.056679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:52827 (local address 127.3.19.67:36073)
0925 09:13:37.056892 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:37.056901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:37.057634 (+ 733us) server_negotiation.cc:407] Connection header received
0925 09:13:37.057795 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:37.057798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:37.057852 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:37.057953 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:37.058543 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.059241 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.061140 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.061973 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.062116 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.062244 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:37.062548 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 09:13:37.062676 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 09:13:37.063140 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.056400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:45725 (local address 127.3.19.66:44065)
0925 09:13:37.057317 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:37.057334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:37.057464 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:37.057855 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:37.057859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:37.057882 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:37.058228 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.058236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.059233 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.059242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.060957 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.060973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.062214 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.062222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.062376 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.062399 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 09:13:37.062510 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 09:13:37.062663 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":7,"thread_start_us":490,"threads_started":1}
I20260925 09:13:37.063162 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.056539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:36073 (local address 127.3.19.66:52827)
0925 09:13:37.057505 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:37.057537 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:37.057669 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:37.057978 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:37.057982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:37.058027 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:37.058398 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.058406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.059393 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.059398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.060993 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.061004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.062246 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.062252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.062397 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.062419 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 09:13:37.062524 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 09:13:37.062677 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":95,"thread_start_us":575,"threads_started":1}
I20260925 09:13:37.064342 5070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566a8002e0d1476fbf7005bb34da7f9d" candidate_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" is_pre_election: true
I20260925 09:13:37.064342 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566a8002e0d1476fbf7005bb34da7f9d" candidate_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b326626ea524046967381848bde308b" is_pre_election: true
I20260925 09:13:37.064958 5350 raft_consensus.cc:2454] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24e3ca0e87a44506a1d8cf084e98cc90 in term 0.
I20260925 09:13:37.064947 5070 raft_consensus.cc:2454] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24e3ca0e87a44506a1d8cf084e98cc90 in term 0.
I20260925 09:13:37.065708 5143 leader_election.cc:302] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b326626ea524046967381848bde308b, 24e3ca0e87a44506a1d8cf084e98cc90, 3d9fb7c7e27548b8a56c8a052dcf5dc0; no voters:
I20260925 09:13:37.066154 5427 raft_consensus.cc:2781] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:13:37.066335 5427 raft_consensus.cc:482] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:13:37.066442 5427 raft_consensus.cc:3037] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:37.068552 5427 raft_consensus.cc:504] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.069157 5427 leader_election.cc:288] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [CANDIDATE]: Term 1 election: Requested vote from peers 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65:45725), 0b326626ea524046967381848bde308b (127.3.19.67:36073)
I20260925 09:13:37.069660 5070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566a8002e0d1476fbf7005bb34da7f9d" candidate_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0"
I20260925 09:13:37.069692 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566a8002e0d1476fbf7005bb34da7f9d" candidate_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b326626ea524046967381848bde308b"
I20260925 09:13:37.069867 5070 raft_consensus.cc:3037] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:37.069936 5350 raft_consensus.cc:3037] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:37.072593 5350 raft_consensus.cc:2454] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24e3ca0e87a44506a1d8cf084e98cc90 in term 1.
I20260925 09:13:37.072593 5070 raft_consensus.cc:2454] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24e3ca0e87a44506a1d8cf084e98cc90 in term 1.
I20260925 09:13:37.072999 5143 leader_election.cc:302] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b326626ea524046967381848bde308b, 24e3ca0e87a44506a1d8cf084e98cc90; no voters:
I20260925 09:13:37.073320 5427 raft_consensus.cc:2781] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:37.073755 5427 raft_consensus.cc:686] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 LEADER]: Becoming Leader. State: Replica: 24e3ca0e87a44506a1d8cf084e98cc90, State: Running, Role: LEADER
I20260925 09:13:37.074283 5427 consensus_queue.cc:237] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.078713 4911 catalog_manager.cc:5495] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24e3ca0e87a44506a1d8cf084e98cc90 (127.3.19.66). New cstate: current_term: 1 leader_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } health_report { overall_health: UNKNOWN } } }
W20260925 09:13:37.089437 5401 tablet.cc:2369] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 09:13:37 dist-test-slave-l6qh krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:37.151888 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.136101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.3.19.126:45681)
0925 09:13:37.136346 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:37.136351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:37.136930 (+ 579us) server_negotiation.cc:407] Connection header received
0925 09:13:37.137175 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:37.137179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:37.137293 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:37.137420 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:37.138755 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.139600 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.141029 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.141589 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.141741 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.145545 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:37.145550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:37.145553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:37.145583 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:37.148534 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:37.149489 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:37.149493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:37.149496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:37.149660 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:37.150143 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:37.150155 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:37.150157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:37.150479 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:37.150669 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:37.151113 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 09:13:37.151510 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":170,"server-negotiator.queue_time_us":86}
I20260925 09:13:37.151917 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.136112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:45681 (local address 127.0.0.1:52712)
0925 09:13:37.136805 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:37.136823 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:37.137016 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:37.137447 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:37.137452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:37.138298 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:37.138582 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.138592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.139737 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.139746 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.140880 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.140892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.141804 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.141811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.141978 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.142767 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:37.142786 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:37.145349 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:37.148680 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:37.148696 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:37.148705 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:37.149322 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:37.149837 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:37.149842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:37.149846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:37.150010 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:37.150638 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:37.150653 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:37.150839 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 09:13:37.151020 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 09:13:37.151356 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":407,"threads_started":1}
I20260925 09:13:37.158473 4911 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:52712:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 09:13:37.159483 4911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:13:37.166478 5050 tablet_service.cc:1467] Processing CreateTablet for tablet 86109cd1c43847c384e4e6475212715e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fcfe30162e14de3b32f5809b959c238]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 09:13:37.166601 5189 tablet_service.cc:1467] Processing CreateTablet for tablet 86109cd1c43847c384e4e6475212715e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fcfe30162e14de3b32f5809b959c238]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 09:13:37.166941 5329 tablet_service.cc:1467] Processing CreateTablet for tablet 86109cd1c43847c384e4e6475212715e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fcfe30162e14de3b32f5809b959c238]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 09:13:37.167524 5189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86109cd1c43847c384e4e6475212715e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:37.167524 5050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86109cd1c43847c384e4e6475212715e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:37.167827 5329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86109cd1c43847c384e4e6475212715e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:37.174167 5419 tablet_bootstrap.cc:492] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b: Bootstrap starting.
I20260925 09:13:37.174618 5421 tablet_bootstrap.cc:492] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90: Bootstrap starting.
I20260925 09:13:37.174676 5420 tablet_bootstrap.cc:492] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Bootstrap starting.
I20260925 09:13:37.176100 5419 tablet_bootstrap.cc:654] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:37.176296 5421 tablet_bootstrap.cc:654] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:37.176473 5420 tablet_bootstrap.cc:654] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:37.178593 5419 tablet_bootstrap.cc:492] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b: No bootstrap required, opened a new log
I20260925 09:13:37.178737 5419 ts_tablet_manager.cc:1397] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260925 09:13:37.179015 5420 tablet_bootstrap.cc:492] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0: No bootstrap required, opened a new log
I20260925 09:13:37.179136 5421 tablet_bootstrap.cc:492] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90: No bootstrap required, opened a new log
I20260925 09:13:37.179152 5420 ts_tablet_manager.cc:1397] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 09:13:37.179324 5421 ts_tablet_manager.cc:1397] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 09:13:37.179572 5419 raft_consensus.cc:348] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.179826 5419 raft_consensus.cc:374] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:37.179929 5419 raft_consensus.cc:729] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b326626ea524046967381848bde308b, State: Initialized, Role: FOLLOWER
I20260925 09:13:37.179927 5420 raft_consensus.cc:348] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.180138 5420 raft_consensus.cc:374] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:37.180227 5420 raft_consensus.cc:729] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d9fb7c7e27548b8a56c8a052dcf5dc0, State: Initialized, Role: FOLLOWER
I20260925 09:13:37.180071 5421 raft_consensus.cc:348] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.180145 5419 consensus_queue.cc:260] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.180342 5421 raft_consensus.cc:374] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:37.180444 5421 raft_consensus.cc:729] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e3ca0e87a44506a1d8cf084e98cc90, State: Initialized, Role: FOLLOWER
I20260925 09:13:37.180424 5420 consensus_queue.cc:260] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.180707 5419 ts_tablet_manager.cc:1428] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 09:13:37.180728 5421 consensus_queue.cc:260] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.180985 5420 ts_tablet_manager.cc:1428] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 09:13:37.181083 5425 tablet_replica.cc:439] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } } }
I20260925 09:13:37.181193 5427 tablet_replica.cc:439] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } } }
I20260925 09:13:37.181416 5425 tablet_replica.cc:442] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:37.181488 5427 tablet_replica.cc:442] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:37.181666 5426 tablet_replica.cc:439] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } } }
I20260925 09:13:37.182024 5426 tablet_replica.cc:442] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:37.182075 5421 ts_tablet_manager.cc:1428] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 09:13:37.439880 5425 raft_consensus.cc:482] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:13:37.440129 5425 raft_consensus.cc:504] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.441854 5425 leader_election.cc:288] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65:45725), 24e3ca0e87a44506a1d8cf084e98cc90 (127.3.19.66:38795)
I20260925 09:13:37.448627 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.442292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:49725 (local address 127.3.19.66:38795)
0925 09:13:37.443473 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:37.443477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:37.443494 (+ 17us) server_negotiation.cc:407] Connection header received
0925 09:13:37.443552 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:37.443555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:37.443608 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:37.443693 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:37.444320 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.445074 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.446873 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.447650 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.447805 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.447920 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:37.448186 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 09:13:37.448319 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086,"thread_start_us":953,"threads_started":1}
I20260925 09:13:37.448724 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.442175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:38795 (local address 127.3.19.67:49725)
0925 09:13:37.443184 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:37.443201 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:37.443304 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:37.443756 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:37.443762 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:37.443790 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:37.444089 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.444135 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.445219 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.445251 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.446713 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.446726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.447824 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.447830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.448017 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.448041 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 09:13:37.448163 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 09:13:37.448304 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":63,"spinlock_wait_cycles":17152,"thread_start_us":745,"threads_started":1}
I20260925 09:13:37.449023 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.442201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:38217 (local address 127.3.19.65:45725)
0925 09:13:37.442449 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:37.442459 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:37.443045 (+ 586us) server_negotiation.cc:407] Connection header received
0925 09:13:37.443406 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:37.443414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:37.443491 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:37.443554 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:37.444431 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.445203 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.447130 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.447966 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.448138 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.448256 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:37.448558 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 09:13:37.448689 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 09:13:37.450078 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.442114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:45725 (local address 127.3.19.67:38217)
0925 09:13:37.442904 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:37.442929 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:37.443167 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:37.443763 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:37.443767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:37.443891 (+ 124us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:37.444245 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.444258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.445430 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.445436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.446966 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.446979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.448201 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.448215 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.448391 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.448414 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 09:13:37.448532 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 09:13:37.448671 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":43,"spinlock_wait_cycles":54912,"thread_start_us":497,"threads_started":1}
I20260925 09:13:37.449859 5210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86109cd1c43847c384e4e6475212715e" candidate_uuid: "0b326626ea524046967381848bde308b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" is_pre_election: true
I20260925 09:13:37.450397 5210 raft_consensus.cc:2454] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b326626ea524046967381848bde308b in term 0.
I20260925 09:13:37.450800 5070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86109cd1c43847c384e4e6475212715e" candidate_uuid: "0b326626ea524046967381848bde308b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" is_pre_election: true
I20260925 09:13:37.451069 5070 raft_consensus.cc:2454] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b326626ea524046967381848bde308b in term 0.
I20260925 09:13:37.451064 5283 leader_election.cc:302] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b326626ea524046967381848bde308b, 24e3ca0e87a44506a1d8cf084e98cc90; no voters:
I20260925 09:13:37.451507 5425 raft_consensus.cc:2781] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:13:37.451638 5425 raft_consensus.cc:482] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:13:37.451740 5425 raft_consensus.cc:3037] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:37.453926 5425 raft_consensus.cc:504] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.454489 5425 leader_election.cc:288] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [CANDIDATE]: Term 1 election: Requested vote from peers 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65:45725), 24e3ca0e87a44506a1d8cf084e98cc90 (127.3.19.66:38795)
I20260925 09:13:37.454989 5070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86109cd1c43847c384e4e6475212715e" candidate_uuid: "0b326626ea524046967381848bde308b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0"
I20260925 09:13:37.455004 5210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86109cd1c43847c384e4e6475212715e" candidate_uuid: "0b326626ea524046967381848bde308b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e3ca0e87a44506a1d8cf084e98cc90"
I20260925 09:13:37.455245 5070 raft_consensus.cc:3037] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:37.455204 5210 raft_consensus.cc:3037] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:37.458587 5210 raft_consensus.cc:2454] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b326626ea524046967381848bde308b in term 1.
I20260925 09:13:37.458598 5070 raft_consensus.cc:2454] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b326626ea524046967381848bde308b in term 1.
I20260925 09:13:37.459107 5282 leader_election.cc:302] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b326626ea524046967381848bde308b, 3d9fb7c7e27548b8a56c8a052dcf5dc0; no voters:
I20260925 09:13:37.459463 5425 raft_consensus.cc:2781] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:37.459839 5425 raft_consensus.cc:686] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [term 1 LEADER]: Becoming Leader. State: Replica: 0b326626ea524046967381848bde308b, State: Running, Role: LEADER
I20260925 09:13:37.460400 5425 consensus_queue.cc:237] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.463967 5444 tablet_replica.cc:439] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b: TxnStatusTablet state changed. Reason: New leader 0b326626ea524046967381848bde308b. Latest consensus state: current_term: 1 leader_uuid: "0b326626ea524046967381848bde308b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } } }
I20260925 09:13:37.464390 5444 tablet_replica.cc:442] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b: This TxnStatusTablet replica's current role is: LEADER
I20260925 09:13:37.465022 5446 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 09:13:37.465291 5446 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 09:13:37.465329 4910 catalog_manager.cc:5495] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b326626ea524046967381848bde308b (127.3.19.67). New cstate: current_term: 1 leader_uuid: "0b326626ea524046967381848bde308b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } health_report { overall_health: HEALTHY } } }
I20260925 09:13:37.547401 5418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.540868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.3.19.67:36073)
0925 09:13:37.541073 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:37.541077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:37.541192 (+ 115us) server_negotiation.cc:407] Connection header received
0925 09:13:37.541513 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:37.541517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:37.541593 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:37.541725 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:13:37.542154 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.542978 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.544949 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.545511 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.545616 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.546189 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:13:37.546802 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:13:37.546925 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:37.547062 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 09:13:37.547138 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 09:13:37.547523 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.540793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:36073 (local address 127.0.0.1:60300)
0925 09:13:37.541069 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:37.541090 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:37.541281 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:37.541772 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:37.541777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:37.541792 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 09:13:37.542004 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.542015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.543113 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.543118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.544783 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.544799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.545687 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.545693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.545913 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.545997 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 09:13:37.546919 (+ 922us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 09:13:37.546945 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 09:13:37.547037 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 09:13:37.547217 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 09:13:37.556433 5210 raft_consensus.cc:1261] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Refusing update from remote peer 0b326626ea524046967381848bde308b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:37.556850 5070 raft_consensus.cc:1261] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Refusing update from remote peer 0b326626ea524046967381848bde308b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:37.557435 5444 consensus_queue.cc:1035] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [LEADER]: Connected to new peer: Peer: permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:37.557876 5425 consensus_queue.cc:1035] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:37.567236 5426 tablet_replica.cc:439] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0: TxnStatusTablet state changed. Reason: New leader 0b326626ea524046967381848bde308b. Latest consensus state: current_term: 1 leader_uuid: "0b326626ea524046967381848bde308b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } } }
I20260925 09:13:37.567660 5426 tablet_replica.cc:442] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:37.572186 5449 mvcc.cc:205] Tried to move back new op lower bound from 7333181921501167616 to 7333181921122320384. Current Snapshot: MvccSnapshot[applied={T|T < 7333181921501167616}]
I20260925 09:13:37.573712 5451 mvcc.cc:205] Tried to move back new op lower bound from 7333181921501167616 to 7333181921122320384. Current Snapshot: MvccSnapshot[applied={T|T < 7333181921501167616}]
I20260925 09:13:37.575039 5444 tablet_replica.cc:439] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b326626ea524046967381848bde308b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } } }
I20260925 09:13:37.575336 5444 tablet_replica.cc:442] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b: This TxnStatusTablet replica's current role is: LEADER
I20260925 09:13:37.579193 5427 tablet_replica.cc:439] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90: TxnStatusTablet state changed. Reason: New leader 0b326626ea524046967381848bde308b. Latest consensus state: current_term: 1 leader_uuid: "0b326626ea524046967381848bde308b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } } }
I20260925 09:13:37.579461 5427 tablet_replica.cc:442] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:37.580554 5425 tablet_replica.cc:439] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b326626ea524046967381848bde308b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } } }
I20260925 09:13:37.580850 5425 tablet_replica.cc:442] T 86109cd1c43847c384e4e6475212715e P 0b326626ea524046967381848bde308b: This TxnStatusTablet replica's current role is: LEADER
I20260925 09:13:37.581522 5426 tablet_replica.cc:439] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b326626ea524046967381848bde308b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } } }
I20260925 09:13:37.581768 5426 tablet_replica.cc:442] T 86109cd1c43847c384e4e6475212715e P 3d9fb7c7e27548b8a56c8a052dcf5dc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:37.589793 5427 tablet_replica.cc:439] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b326626ea524046967381848bde308b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } } }
I20260925 09:13:37.590118 5427 tablet_replica.cc:442] T 86109cd1c43847c384e4e6475212715e P 24e3ca0e87a44506a1d8cf084e98cc90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:13:37.595059 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.583270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.3.19.66:38795)
0925 09:13:37.583493 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:37.583502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:37.587780 (+ 4278us) server_negotiation.cc:407] Connection header received
0925 09:13:37.588102 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:37.588108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:37.588226 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:37.588364 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:13:37.589015 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.590187 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.592756 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.593192 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.593363 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.593840 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:13:37.594350 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:13:37.594505 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:37.594611 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 09:13:37.594705 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 09:13:37.611936 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.605633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:36073 (local address 127.3.19.66:54815)
0925 09:13:37.606336 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:37.606352 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:37.606475 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:37.606869 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:37.606873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:37.606882 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 09:13:37.607116 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.607123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.608573 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.608578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.609805 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.609818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.610840 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.610845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.610960 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.611020 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 09:13:37.611452 (+ 432us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 09:13:37.611460 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 09:13:37.611522 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 09:13:37.611643 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":366,"threads_started":1}
I20260925 09:13:37.611975 5418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.605641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:54815 (local address 127.3.19.67:36073)
0925 09:13:37.605854 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:37.605860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:37.606442 (+ 582us) server_negotiation.cc:407] Connection header received
0925 09:13:37.606637 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:37.606643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:37.606731 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:37.606851 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:13:37.607250 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.608416 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.609955 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.610650 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.610779 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.611225 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:13:37.611354 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:13:37.611445 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:37.611568 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 09:13:37.611653 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 09:13:37.623176 5069 raft_consensus.cc:1261] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Refusing update from remote peer 24e3ca0e87a44506a1d8cf084e98cc90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:37.623574 5350 raft_consensus.cc:1261] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Refusing update from remote peer 24e3ca0e87a44506a1d8cf084e98cc90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:37.623858 5427 consensus_queue.cc:1035] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:37.624436 5427 consensus_queue.cc:1035] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:37.639999 5450 mvcc.cc:205] Tried to move back new op lower bound from 7333181921777823744 to 7333181919540051968. Current Snapshot: MvccSnapshot[applied={T|T < 7333181921777823744}]
I20260925 09:13:37.676450 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.668902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:38795 (local address 127.3.19.67:36209)
0925 09:13:37.669734 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:37.669751 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:37.669919 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:37.670478 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:37.670483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:37.670497 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 09:13:37.670903 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.670916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.672413 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.672417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.674180 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:37.674192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.675156 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.675164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:37.675343 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.675430 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 09:13:37.675931 (+ 501us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 09:13:37.675947 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 09:13:37.676014 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 09:13:37.676124 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":510,"threads_started":1}
I20260925 09:13:37.676437 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:37.668879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:36209 (local address 127.3.19.66:38795)
0925 09:13:37.669074 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:37.669081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:37.669841 (+ 760us) server_negotiation.cc:407] Connection header received
0925 09:13:37.670061 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:37.670067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:37.670184 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:37.670291 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:13:37.671109 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.672279 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.674323 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:37.674975 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:37.675116 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:37.675660 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:13:37.675818 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:13:37.675919 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:37.676061 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 09:13:37.676150 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 09:13:37.716437 4911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:13:37.716750 4911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:13:37.721341 4911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c: Sending DeleteTablet for 3 replicas of tablet 566a8002e0d1476fbf7005bb34da7f9d
I20260925 09:13:37.722564 5050 tablet_service.cc:1514] Processing DeleteTablet for tablet 566a8002e0d1476fbf7005bb34da7f9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:37 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:51348
I20260925 09:13:37.723476 5471 tablet_replica.cc:330] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0: stopping tablet replica
I20260925 09:13:37.723938 5471 raft_consensus.cc:2229] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:13:37.724054 5189 tablet_service.cc:1514] Processing DeleteTablet for tablet 566a8002e0d1476fbf7005bb34da7f9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:37 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:36490
I20260925 09:13:37.724324 5471 raft_consensus.cc:2258] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:37.726354 5471 ts_tablet_manager.cc:1905] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:37.728018 5329 tablet_service.cc:1514] Processing DeleteTablet for tablet 566a8002e0d1476fbf7005bb34da7f9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:37 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:60296
I20260925 09:13:37.729750 4910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:13:37.730476 4910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:13:37.733820 5473 tablet_replica.cc:330] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b: stopping tablet replica
I20260925 09:13:37.738111 5472 tablet_replica.cc:330] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90: stopping tablet replica
I20260925 09:13:37.739993 5472 raft_consensus.cc:2229] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:13:37.739993 5050 tablet_service.cc:1467] Processing CreateTablet for tablet 468ea0f9e4734d14a7f4e0f491c113e3 (DEFAULT_TABLE table=test-table [id=fd58f157969446c885a32b29dc1db6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:37.740458 5050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468ea0f9e4734d14a7f4e0f491c113e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:37.740875 5472 raft_consensus.cc:2258] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:37.744191 5472 ts_tablet_manager.cc:1905] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:37.740001 5329 tablet_service.cc:1467] Processing CreateTablet for tablet 468ea0f9e4734d14a7f4e0f491c113e3 (DEFAULT_TABLE table=test-table [id=fd58f157969446c885a32b29dc1db6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:37.745639 5329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468ea0f9e4734d14a7f4e0f491c113e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:37.745893 5189 tablet_service.cc:1467] Processing CreateTablet for tablet 468ea0f9e4734d14a7f4e0f491c113e3 (DEFAULT_TABLE table=test-table [id=fd58f157969446c885a32b29dc1db6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:37.746347 5189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468ea0f9e4734d14a7f4e0f491c113e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:37.752617 5473 raft_consensus.cc:2229] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:13:37.752993 5473 raft_consensus.cc:2258] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:37.758431 5471 ts_tablet_manager.cc:1918] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:13:37.758656 5471 log.cc:1196] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/566a8002e0d1476fbf7005bb34da7f9d
I20260925 09:13:37.759121 5475 tablet_bootstrap.cc:492] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Bootstrap starting.
I20260925 09:13:37.759379 5471 ts_tablet_manager.cc:1939] T 566a8002e0d1476fbf7005bb34da7f9d P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Deleting consensus metadata
I20260925 09:13:37.761101 5475 tablet_bootstrap.cc:654] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:37.761415 4898 catalog_manager.cc:4841] TS 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65:45725): tablet 566a8002e0d1476fbf7005bb34da7f9d (table test-table [id=7dc3b6ecc1124fb4954e2a0256a28b98]) successfully deleted
I20260925 09:13:37.762653 5473 ts_tablet_manager.cc:1905] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:37.766884 5472 ts_tablet_manager.cc:1918] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:13:37.767203 5472 log.cc:1196] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/566a8002e0d1476fbf7005bb34da7f9d
I20260925 09:13:37.767827 5474 tablet_bootstrap.cc:492] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b: Bootstrap starting.
I20260925 09:13:37.768570 5472 ts_tablet_manager.cc:1939] T 566a8002e0d1476fbf7005bb34da7f9d P 24e3ca0e87a44506a1d8cf084e98cc90: Deleting consensus metadata
I20260925 09:13:37.770975 5474 tablet_bootstrap.cc:654] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:37.773396 4899 catalog_manager.cc:4841] TS 24e3ca0e87a44506a1d8cf084e98cc90 (127.3.19.66:38795): tablet 566a8002e0d1476fbf7005bb34da7f9d (table test-table [id=7dc3b6ecc1124fb4954e2a0256a28b98]) successfully deleted
I20260925 09:13:37.784700 5477 tablet_bootstrap.cc:492] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90: Bootstrap starting.
I20260925 09:13:37.781502 5473 ts_tablet_manager.cc:1918] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:13:37.785800 5473 log.cc:1196] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/566a8002e0d1476fbf7005bb34da7f9d
I20260925 09:13:37.786688 5473 ts_tablet_manager.cc:1939] T 566a8002e0d1476fbf7005bb34da7f9d P 0b326626ea524046967381848bde308b: Deleting consensus metadata
I20260925 09:13:37.786713 5475 tablet_bootstrap.cc:492] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: No bootstrap required, opened a new log
I20260925 09:13:37.786890 5475 ts_tablet_manager.cc:1397] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.005s
I20260925 09:13:37.786922 5477 tablet_bootstrap.cc:654] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:37.787856 5475 raft_consensus.cc:348] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.788519 5475 raft_consensus.cc:374] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:37.788637 5475 raft_consensus.cc:729] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d9fb7c7e27548b8a56c8a052dcf5dc0, State: Initialized, Role: FOLLOWER
I20260925 09:13:37.788938 5475 consensus_queue.cc:260] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.790728 4899 catalog_manager.cc:4841] TS 0b326626ea524046967381848bde308b (127.3.19.67:36073): tablet 566a8002e0d1476fbf7005bb34da7f9d (table test-table [id=7dc3b6ecc1124fb4954e2a0256a28b98]) successfully deleted
I20260925 09:13:37.791985 5475 ts_tablet_manager.cc:1428] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260925 09:13:37.793154 5474 tablet_bootstrap.cc:492] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b: No bootstrap required, opened a new log
I20260925 09:13:37.793951 5474 ts_tablet_manager.cc:1397] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.004s
I20260925 09:13:37.795430 5474 raft_consensus.cc:348] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.795864 5474 raft_consensus.cc:374] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:37.796077 5474 raft_consensus.cc:729] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b326626ea524046967381848bde308b, State: Initialized, Role: FOLLOWER
I20260925 09:13:37.796485 5474 consensus_queue.cc:260] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.798866 5477 tablet_bootstrap.cc:492] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90: No bootstrap required, opened a new log
I20260925 09:13:37.799057 5477 ts_tablet_manager.cc:1397] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260925 09:13:37.799873 5477 raft_consensus.cc:348] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.800117 5477 raft_consensus.cc:374] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:37.800227 5477 raft_consensus.cc:729] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e3ca0e87a44506a1d8cf084e98cc90, State: Initialized, Role: FOLLOWER
I20260925 09:13:37.800388 5474 ts_tablet_manager.cc:1428] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260925 09:13:37.800520 5477 consensus_queue.cc:260] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:37.802878 5477 ts_tablet_manager.cc:1428] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 09:13:38.070711 5466 raft_consensus.cc:482] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:13:38.070967 5466 raft_consensus.cc:504] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:38.071892 5466 leader_election.cc:288] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65:45725), 0b326626ea524046967381848bde308b (127.3.19.67:36073)
I20260925 09:13:38.072515 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468ea0f9e4734d14a7f4e0f491c113e3" candidate_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b326626ea524046967381848bde308b" is_pre_election: true
I20260925 09:13:38.072474 5069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468ea0f9e4734d14a7f4e0f491c113e3" candidate_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" is_pre_election: true
I20260925 09:13:38.072773 5350 raft_consensus.cc:2454] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24e3ca0e87a44506a1d8cf084e98cc90 in term 0.
I20260925 09:13:38.072814 5069 raft_consensus.cc:2454] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24e3ca0e87a44506a1d8cf084e98cc90 in term 0.
I20260925 09:13:38.073350 5142 leader_election.cc:302] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24e3ca0e87a44506a1d8cf084e98cc90, 3d9fb7c7e27548b8a56c8a052dcf5dc0; no voters:
I20260925 09:13:38.073664 5466 raft_consensus.cc:2781] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:13:38.073796 5466 raft_consensus.cc:482] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:13:38.073889 5466 raft_consensus.cc:3037] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:38.077347 5466 raft_consensus.cc:504] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:38.078153 5466 leader_election.cc:288] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [CANDIDATE]: Term 1 election: Requested vote from peers 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65:45725), 0b326626ea524046967381848bde308b (127.3.19.67:36073)
I20260925 09:13:38.078750 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468ea0f9e4734d14a7f4e0f491c113e3" candidate_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b326626ea524046967381848bde308b"
I20260925 09:13:38.078742 5069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468ea0f9e4734d14a7f4e0f491c113e3" candidate_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0"
I20260925 09:13:38.078995 5350 raft_consensus.cc:3037] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:38.079033 5069 raft_consensus.cc:3037] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:38.080957 5350 raft_consensus.cc:2454] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24e3ca0e87a44506a1d8cf084e98cc90 in term 1.
I20260925 09:13:38.081537 5143 leader_election.cc:302] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b326626ea524046967381848bde308b, 24e3ca0e87a44506a1d8cf084e98cc90; no voters:
I20260925 09:13:38.081862 5466 raft_consensus.cc:2781] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:38.082085 5069 raft_consensus.cc:2454] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24e3ca0e87a44506a1d8cf084e98cc90 in term 1.
I20260925 09:13:38.082175 5466 raft_consensus.cc:686] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 LEADER]: Becoming Leader. State: Replica: 24e3ca0e87a44506a1d8cf084e98cc90, State: Running, Role: LEADER
I20260925 09:13:38.082567 5466 consensus_queue.cc:237] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:38.085726 4911 catalog_manager.cc:5495] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24e3ca0e87a44506a1d8cf084e98cc90 (127.3.19.66). New cstate: current_term: 1 leader_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } health_report { overall_health: UNKNOWN } } }
I20260925 09:13:38.135705 5350 raft_consensus.cc:1261] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Refusing update from remote peer 24e3ca0e87a44506a1d8cf084e98cc90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:38.136055 5069 raft_consensus.cc:1261] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Refusing update from remote peer 24e3ca0e87a44506a1d8cf084e98cc90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:38.136332 5466 consensus_queue.cc:1035] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:38.136679 5463 consensus_queue.cc:1035] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 09:13:38.176056 5450 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 09:13:39.766635 4911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:13:39.766863 4911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:13:39.771133 4911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c: Sending DeleteTablet for 3 replicas of tablet 468ea0f9e4734d14a7f4e0f491c113e3
I20260925 09:13:39.771986 5050 tablet_service.cc:1514] Processing DeleteTablet for tablet 468ea0f9e4734d14a7f4e0f491c113e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:39 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:51348
I20260925 09:13:39.772142 5189 tablet_service.cc:1514] Processing DeleteTablet for tablet 468ea0f9e4734d14a7f4e0f491c113e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:39 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:36490
I20260925 09:13:39.772693 5329 tablet_service.cc:1514] Processing DeleteTablet for tablet 468ea0f9e4734d14a7f4e0f491c113e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:39 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:60296
I20260925 09:13:39.782902 5506 tablet_replica.cc:330] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: stopping tablet replica
I20260925 09:13:39.783318 5506 raft_consensus.cc:2229] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:13:39.783744 5506 raft_consensus.cc:2258] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:39.784107 5507 tablet_replica.cc:330] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90: stopping tablet replica
I20260925 09:13:39.784382 5507 raft_consensus.cc:2229] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:13:39.784934 5506 ts_tablet_manager.cc:1905] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:39.785672 5508 tablet_replica.cc:330] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b: stopping tablet replica
I20260925 09:13:39.785655 5507 raft_consensus.cc:2258] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:39.785931 5508 raft_consensus.cc:2229] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:13:39.786192 5508 raft_consensus.cc:2258] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:39.786713 5507 ts_tablet_manager.cc:1905] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:39.787176 5508 ts_tablet_manager.cc:1905] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 09:13:39 dist-test-slave-l6qh krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327619, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 09:13:39 dist-test-slave-l6qh krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327619, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:39.879122 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:39.860841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.3.19.126:45681)
0925 09:13:39.861886 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:39.861890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:39.861918 (+ 28us) server_negotiation.cc:407] Connection header received
0925 09:13:39.861976 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:39.861980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:39.862040 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:39.862201 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:39.863689 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:39.864623 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:39.866443 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:39.866941 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:39.867053 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:39.871258 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:39.871264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:39.871268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:39.871322 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:39.876216 (+ 4894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:39.877014 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:39.877018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:39.877021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:39.877163 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:39.877620 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:39.877625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:39.877635 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:39.877934 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:39.878204 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:39.878515 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 09:13:39.878746 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":749,"threads_started":1}
I20260925 09:13:39.882990 4911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:13:39.883538 4911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:13:39.896745 5506 ts_tablet_manager.cc:1918] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 09:13:39.896900 5506 log.cc:1196] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/468ea0f9e4734d14a7f4e0f491c113e3
I20260925 09:13:39.897576 5508 ts_tablet_manager.cc:1918] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 09:13:39.897734 5508 log.cc:1196] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/468ea0f9e4734d14a7f4e0f491c113e3
I20260925 09:13:39.898051 5507 ts_tablet_manager.cc:1918] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 09:13:39.898226 5507 log.cc:1196] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/468ea0f9e4734d14a7f4e0f491c113e3
I20260925 09:13:39.901844 5050 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9f49f55c3b4092accc2fa6d2d949f5 (DEFAULT_TABLE table=test-table [id=aee949ecc6c24583b978af729b927234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:39.902148 5189 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9f49f55c3b4092accc2fa6d2d949f5 (DEFAULT_TABLE table=test-table [id=aee949ecc6c24583b978af729b927234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:39.902278 5050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9f49f55c3b4092accc2fa6d2d949f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:39.902617 5189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9f49f55c3b4092accc2fa6d2d949f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:39.902933 5329 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9f49f55c3b4092accc2fa6d2d949f5 (DEFAULT_TABLE table=test-table [id=aee949ecc6c24583b978af729b927234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:39.903367 5329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9f49f55c3b4092accc2fa6d2d949f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:39.911011 5519 tablet_bootstrap.cc:492] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Bootstrap starting.
I20260925 09:13:39.911755 5517 tablet_bootstrap.cc:492] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90: Bootstrap starting.
I20260925 09:13:39.913023 5519 tablet_bootstrap.cc:654] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:39.913519 5518 tablet_bootstrap.cc:492] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b: Bootstrap starting.
I20260925 09:13:39.913790 5517 tablet_bootstrap.cc:654] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:39.915267 5518 tablet_bootstrap.cc:654] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:39.916411 5517 tablet_bootstrap.cc:492] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90: No bootstrap required, opened a new log
I20260925 09:13:39.916563 5517 ts_tablet_manager.cc:1397] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 09:13:39.917208 5517 raft_consensus.cc:348] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:39.917500 5517 raft_consensus.cc:374] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:39.917595 5517 raft_consensus.cc:729] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e3ca0e87a44506a1d8cf084e98cc90, State: Initialized, Role: FOLLOWER
I20260925 09:13:39.917961 5519 tablet_bootstrap.cc:492] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: No bootstrap required, opened a new log
I20260925 09:13:39.917891 5517 consensus_queue.cc:260] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:39.918150 5519 ts_tablet_manager.cc:1397] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 09:13:39.918435 5517 ts_tablet_manager.cc:1428] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 09:13:39.918862 5519 raft_consensus.cc:348] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:39.919226 5519 raft_consensus.cc:374] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:39.919387 5519 raft_consensus.cc:729] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d9fb7c7e27548b8a56c8a052dcf5dc0, State: Initialized, Role: FOLLOWER
I20260925 09:13:39.919771 5518 tablet_bootstrap.cc:492] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b: No bootstrap required, opened a new log
I20260925 09:13:39.919672 5519 consensus_queue.cc:260] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:39.920011 5518 ts_tablet_manager.cc:1397] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260925 09:13:39.920734 5519 ts_tablet_manager.cc:1428] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 09:13:39.921057 5518 raft_consensus.cc:348] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:39.921404 5518 raft_consensus.cc:374] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:39.921551 5518 raft_consensus.cc:729] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b326626ea524046967381848bde308b, State: Initialized, Role: FOLLOWER
I20260925 09:13:39.921864 5518 consensus_queue.cc:260] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:39.922432 5518 ts_tablet_manager.cc:1428] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 09:13:39.939793 5506 ts_tablet_manager.cc:1939] T 468ea0f9e4734d14a7f4e0f491c113e3 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Deleting consensus metadata
I20260925 09:13:39.941030 4898 catalog_manager.cc:4841] TS 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65:45725): tablet 468ea0f9e4734d14a7f4e0f491c113e3 (table test-table [id=fd58f157969446c885a32b29dc1db6dd]) successfully deleted
I20260925 09:13:39.972915 5507 ts_tablet_manager.cc:1939] T 468ea0f9e4734d14a7f4e0f491c113e3 P 24e3ca0e87a44506a1d8cf084e98cc90: Deleting consensus metadata
I20260925 09:13:39.972916 5508 ts_tablet_manager.cc:1939] T 468ea0f9e4734d14a7f4e0f491c113e3 P 0b326626ea524046967381848bde308b: Deleting consensus metadata
I20260925 09:13:39.974133 4899 catalog_manager.cc:4841] TS 0b326626ea524046967381848bde308b (127.3.19.67:36073): tablet 468ea0f9e4734d14a7f4e0f491c113e3 (table test-table [id=fd58f157969446c885a32b29dc1db6dd]) successfully deleted
I20260925 09:13:39.974413 4899 catalog_manager.cc:4841] TS 24e3ca0e87a44506a1d8cf084e98cc90 (127.3.19.66:38795): tablet 468ea0f9e4734d14a7f4e0f491c113e3 (table test-table [id=fd58f157969446c885a32b29dc1db6dd]) successfully deleted
I20260925 09:13:40.301505 5500 raft_consensus.cc:482] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:13:40.301755 5500 raft_consensus.cc:504] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:40.302488 5500 leader_election.cc:288] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65:45725), 0b326626ea524046967381848bde308b (127.3.19.67:36073)
I20260925 09:13:40.303035 5069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9f49f55c3b4092accc2fa6d2d949f5" candidate_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" is_pre_election: true
I20260925 09:13:40.303094 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9f49f55c3b4092accc2fa6d2d949f5" candidate_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b326626ea524046967381848bde308b" is_pre_election: true
I20260925 09:13:40.303328 5069 raft_consensus.cc:2454] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24e3ca0e87a44506a1d8cf084e98cc90 in term 0.
I20260925 09:13:40.303308 5349 raft_consensus.cc:2454] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24e3ca0e87a44506a1d8cf084e98cc90 in term 0.
I20260925 09:13:40.303833 5143 leader_election.cc:302] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b326626ea524046967381848bde308b, 24e3ca0e87a44506a1d8cf084e98cc90; no voters:
I20260925 09:13:40.304171 5500 raft_consensus.cc:2781] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:13:40.304346 5500 raft_consensus.cc:482] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:13:40.304446 5500 raft_consensus.cc:3037] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:40.400100 5500 raft_consensus.cc:504] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:40.400916 5500 leader_election.cc:288] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [CANDIDATE]: Term 1 election: Requested vote from peers 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65:45725), 0b326626ea524046967381848bde308b (127.3.19.67:36073)
I20260925 09:13:40.401516 5069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9f49f55c3b4092accc2fa6d2d949f5" candidate_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0"
I20260925 09:13:40.401561 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9f49f55c3b4092accc2fa6d2d949f5" candidate_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b326626ea524046967381848bde308b"
I20260925 09:13:40.401751 5349 raft_consensus.cc:3037] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:40.401770 5069 raft_consensus.cc:3037] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:40.500811 5069 raft_consensus.cc:2454] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24e3ca0e87a44506a1d8cf084e98cc90 in term 1.
I20260925 09:13:40.500808 5349 raft_consensus.cc:2454] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24e3ca0e87a44506a1d8cf084e98cc90 in term 1.
I20260925 09:13:40.501472 5143 leader_election.cc:302] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b326626ea524046967381848bde308b, 24e3ca0e87a44506a1d8cf084e98cc90, 3d9fb7c7e27548b8a56c8a052dcf5dc0; no voters:
I20260925 09:13:40.501904 5500 raft_consensus.cc:2781] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:40.502403 5500 raft_consensus.cc:686] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 LEADER]: Becoming Leader. State: Replica: 24e3ca0e87a44506a1d8cf084e98cc90, State: Running, Role: LEADER
I20260925 09:13:40.502800 5500 consensus_queue.cc:237] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:40.507495 4911 catalog_manager.cc:5495] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24e3ca0e87a44506a1d8cf084e98cc90 (127.3.19.66). New cstate: current_term: 1 leader_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } health_report { overall_health: UNKNOWN } } }
I20260925 09:13:40.627681 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:40.622330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45774 (local address 127.3.19.66:38795)
0925 09:13:40.623184 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:40.623188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:40.623208 (+ 20us) server_negotiation.cc:407] Connection header received
0925 09:13:40.623263 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:40.623267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:40.623328 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:40.623410 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:13:40.623815 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.624524 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:40.625881 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.626268 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:40.626376 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:40.626752 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:13:40.626864 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:13:40.626932 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:40.627053 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 09:13:40.627115 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":640,"threads_started":1}
I20260925 09:13:40.636834 5069 raft_consensus.cc:1261] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Refusing update from remote peer 24e3ca0e87a44506a1d8cf084e98cc90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:40.636842 5349 raft_consensus.cc:1261] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Refusing update from remote peer 24e3ca0e87a44506a1d8cf084e98cc90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:40.637370 5500 consensus_queue.cc:1035] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:40.637686 5524 consensus_queue.cc:1035] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:40.695655 4911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:13:40.696018 4911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:13:40.699538 4911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c: Sending DeleteTablet for 3 replicas of tablet 2d9f49f55c3b4092accc2fa6d2d949f5
I20260925 09:13:40.700527 5050 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d9f49f55c3b4092accc2fa6d2d949f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:40 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:51348
I20260925 09:13:40.700632 5189 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d9f49f55c3b4092accc2fa6d2d949f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:40 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:36490
I20260925 09:13:40.700887 5329 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d9f49f55c3b4092accc2fa6d2d949f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:40 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:60296
I20260925 09:13:40.701759 5548 tablet_replica.cc:330] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b: stopping tablet replica
I20260925 09:13:40.701954 5547 tablet_replica.cc:330] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90: stopping tablet replica
I20260925 09:13:40.701990 5548 raft_consensus.cc:2229] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:13:40.701880 4911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 09:13:40.702160 5547 raft_consensus.cc:2229] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:13:40.702298 5548 raft_consensus.cc:2258] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:40.702579 5547 raft_consensus.cc:2258] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 09:13:40.702684 4911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:13:40.703471 5548 ts_tablet_manager.cc:1905] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:40.703536 5547 ts_tablet_manager.cc:1905] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:40.707325 5546 tablet_replica.cc:330] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: stopping tablet replica
I20260925 09:13:40.707559 5546 raft_consensus.cc:2229] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:13:40.707840 5546 raft_consensus.cc:2258] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:40.709113 5546 ts_tablet_manager.cc:1905] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:40.712654 5189 tablet_service.cc:1467] Processing CreateTablet for tablet 5648c0a24a8d44c38da37f0f4a78a7fc (DEFAULT_TABLE table=test-table [id=5409ebd7212b4a7d89035d80269bbe3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:40.712787 5329 tablet_service.cc:1467] Processing CreateTablet for tablet 5648c0a24a8d44c38da37f0f4a78a7fc (DEFAULT_TABLE table=test-table [id=5409ebd7212b4a7d89035d80269bbe3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:40.712765 5050 tablet_service.cc:1467] Processing CreateTablet for tablet 5648c0a24a8d44c38da37f0f4a78a7fc (DEFAULT_TABLE table=test-table [id=5409ebd7212b4a7d89035d80269bbe3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:13:40.714618 5329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5648c0a24a8d44c38da37f0f4a78a7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:40.717873 5548 ts_tablet_manager.cc:1918] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:13:40.717960 5050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5648c0a24a8d44c38da37f0f4a78a7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:40.718086 5548 log.cc:1196] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/2d9f49f55c3b4092accc2fa6d2d949f5
I20260925 09:13:40.718323 5189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5648c0a24a8d44c38da37f0f4a78a7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:40.718647 5548 ts_tablet_manager.cc:1939] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 0b326626ea524046967381848bde308b: Deleting consensus metadata
I20260925 09:13:40.721836 5549 tablet_bootstrap.cc:492] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b: Bootstrap starting.
I20260925 09:13:40.723260 5549 tablet_bootstrap.cc:654] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:40.725412 5549 tablet_bootstrap.cc:492] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b: No bootstrap required, opened a new log
I20260925 09:13:40.725425 5551 tablet_bootstrap.cc:492] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Bootstrap starting.
I20260925 09:13:40.725616 5549 ts_tablet_manager.cc:1397] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 09:13:40.726459 5549 raft_consensus.cc:348] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:40.726866 5549 raft_consensus.cc:374] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:40.726994 5549 raft_consensus.cc:729] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b326626ea524046967381848bde308b, State: Initialized, Role: FOLLOWER
I20260925 09:13:40.727331 5549 consensus_queue.cc:260] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:40.728082 5551 tablet_bootstrap.cc:654] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:40.728637 5549 ts_tablet_manager.cc:1428] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 09:13:40.730517 5551 tablet_bootstrap.cc:492] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0: No bootstrap required, opened a new log
I20260925 09:13:40.730691 5551 ts_tablet_manager.cc:1397] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260925 09:13:40.731407 5551 raft_consensus.cc:348] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:40.731640 5551 raft_consensus.cc:374] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:40.731731 5551 raft_consensus.cc:729] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d9fb7c7e27548b8a56c8a052dcf5dc0, State: Initialized, Role: FOLLOWER
I20260925 09:13:40.732002 5551 consensus_queue.cc:260] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:40.732594 5551 ts_tablet_manager.cc:1428] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 09:13:40.779412 5541 raft_consensus.cc:482] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:13:40.779696 5541 raft_consensus.cc:504] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:40.781399 5541 leader_election.cc:288] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24e3ca0e87a44506a1d8cf084e98cc90 (127.3.19.66:38795), 0b326626ea524046967381848bde308b (127.3.19.67:36073)
I20260925 09:13:40.792728 4899 catalog_manager.cc:4841] TS 0b326626ea524046967381848bde308b (127.3.19.67:36073): tablet 2d9f49f55c3b4092accc2fa6d2d949f5 (table test-table [id=aee949ecc6c24583b978af729b927234]) successfully deleted
I20260925 09:13:40.796975 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:40.781702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:41127 (local address 127.3.19.66:38795)
0925 09:13:40.781924 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:40.781930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:40.783161 (+ 1231us) server_negotiation.cc:407] Connection header received
0925 09:13:40.783415 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:40.783425 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:40.783535 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:40.783606 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:40.784810 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.786395 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:40.792755 (+ 6360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.794630 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:40.794777 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:40.794978 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:40.796371 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0925 09:13:40.796601 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 09:13:40.798358 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:40.781761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:38795 (local address 127.3.19.65:41127)
0925 09:13:40.783023 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:40.783044 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:40.783189 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:40.783893 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:40.783900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:40.783938 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:40.784530 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:40.784551 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.786669 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:40.786677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:40.792454 (+ 5777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:40.792484 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.795780 (+ 3296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:40.795792 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:40.796074 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:40.796131 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 09:13:40.797602 (+ 1471us) client_negotiation.cc:240] Negotiation successful
0925 09:13:40.797836 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"thread_start_us":974,"threads_started":1}
W20260925 09:13:40.801158 5002 leader_election.cc:341] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 24e3ca0e87a44506a1d8cf084e98cc90 (127.3.19.66:38795): Not found: Tablet not found: 5648c0a24a8d44c38da37f0f4a78a7fc
I20260925 09:13:40.805593 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:40.783441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:54467 (local address 127.3.19.67:36073)
0925 09:13:40.800299 (+ 16858us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:40.800303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:40.800331 (+ 28us) server_negotiation.cc:407] Connection header received
0925 09:13:40.800390 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:40.800393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:40.800517 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:40.800634 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:13:40.801184 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.802176 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:40.803755 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.804533 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:40.804679 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:40.804801 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:40.805062 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 09:13:40.805191 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16718,"thread_start_us":16595,"threads_started":1}
I20260925 09:13:40.805711 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:40.783223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:36073 (local address 127.3.19.65:54467)
0925 09:13:40.798954 (+ 15731us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:40.799008 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:40.799162 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:40.800818 (+ 1656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:40.800832 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:40.800865 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:13:40.801020 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:40.801028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.802306 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:40.802314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:40.803592 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:40.803606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.804701 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:40.804707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:40.804900 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:40.804924 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 09:13:40.805027 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 09:13:40.805176 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15571,"thread_start_us":16285,"threads_started":1}
I20260925 09:13:40.806797 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5648c0a24a8d44c38da37f0f4a78a7fc" candidate_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b326626ea524046967381848bde308b" is_pre_election: true
I20260925 09:13:40.807061 5349 raft_consensus.cc:2454] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d9fb7c7e27548b8a56c8a052dcf5dc0 in term 0.
I20260925 09:13:40.807487 5002 leader_election.cc:302] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b326626ea524046967381848bde308b, 3d9fb7c7e27548b8a56c8a052dcf5dc0; no voters: 24e3ca0e87a44506a1d8cf084e98cc90
I20260925 09:13:40.807911 5541 raft_consensus.cc:2781] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:13:40.808020 5541 raft_consensus.cc:482] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:13:40.808136 5541 raft_consensus.cc:3037] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:40.813158 5546 ts_tablet_manager.cc:1918] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:13:40.813179 5547 ts_tablet_manager.cc:1918] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:13:40.813081 5541 raft_consensus.cc:504] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:40.813366 5546 log.cc:1196] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/2d9f49f55c3b4092accc2fa6d2d949f5
I20260925 09:13:40.813365 5547 log.cc:1196] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/2d9f49f55c3b4092accc2fa6d2d949f5
I20260925 09:13:40.813846 5547 ts_tablet_manager.cc:1939] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 24e3ca0e87a44506a1d8cf084e98cc90: Deleting consensus metadata
I20260925 09:13:40.813884 5546 ts_tablet_manager.cc:1939] T 2d9f49f55c3b4092accc2fa6d2d949f5 P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Deleting consensus metadata
I20260925 09:13:40.813997 5541 leader_election.cc:288] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [CANDIDATE]: Term 1 election: Requested vote from peers 24e3ca0e87a44506a1d8cf084e98cc90 (127.3.19.66:38795), 0b326626ea524046967381848bde308b (127.3.19.67:36073)
W20260925 09:13:40.815156 5002 leader_election.cc:341] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 24e3ca0e87a44506a1d8cf084e98cc90 (127.3.19.66:38795): Not found: Tablet not found: 5648c0a24a8d44c38da37f0f4a78a7fc
I20260925 09:13:40.815158 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5648c0a24a8d44c38da37f0f4a78a7fc" candidate_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b326626ea524046967381848bde308b"
I20260925 09:13:40.815408 5349 raft_consensus.cc:3037] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:40.818400 5556 tablet_bootstrap.cc:492] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90: Bootstrap starting.
I20260925 09:13:40.818691 4899 catalog_manager.cc:4841] TS 24e3ca0e87a44506a1d8cf084e98cc90 (127.3.19.66:38795): tablet 2d9f49f55c3b4092accc2fa6d2d949f5 (table test-table [id=aee949ecc6c24583b978af729b927234]) successfully deleted
I20260925 09:13:40.820040 5556 tablet_bootstrap.cc:654] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:40.822371 5556 tablet_bootstrap.cc:492] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90: No bootstrap required, opened a new log
I20260925 09:13:40.822535 5556 ts_tablet_manager.cc:1397] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 09:13:40.823249 4898 catalog_manager.cc:4841] TS 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65:45725): tablet 2d9f49f55c3b4092accc2fa6d2d949f5 (table test-table [id=aee949ecc6c24583b978af729b927234]) successfully deleted
I20260925 09:13:40.823307 5556 raft_consensus.cc:348] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:40.823575 5556 raft_consensus.cc:374] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:40.823666 5556 raft_consensus.cc:729] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e3ca0e87a44506a1d8cf084e98cc90, State: Initialized, Role: FOLLOWER
I20260925 09:13:40.823922 5556 consensus_queue.cc:260] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:40.824473 5556 ts_tablet_manager.cc:1428] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 09:13:40.912937 5349 raft_consensus.cc:2454] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d9fb7c7e27548b8a56c8a052dcf5dc0 in term 1.
I20260925 09:13:40.913659 5002 leader_election.cc:302] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b326626ea524046967381848bde308b, 3d9fb7c7e27548b8a56c8a052dcf5dc0; no voters: 24e3ca0e87a44506a1d8cf084e98cc90
I20260925 09:13:40.914048 5541 raft_consensus.cc:2781] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:40.914750 5541 raft_consensus.cc:686] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 LEADER]: Becoming Leader. State: Replica: 3d9fb7c7e27548b8a56c8a052dcf5dc0, State: Running, Role: LEADER
I20260925 09:13:40.915202 5541 consensus_queue.cc:237] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } }
I20260925 09:13:40.919281 4912 catalog_manager.cc:5495] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65). New cstate: current_term: 1 leader_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9fb7c7e27548b8a56c8a052dcf5dc0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 } health_report { overall_health: UNKNOWN } } }
I20260925 09:13:40.985607 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:40.979548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.3.19.65:45725)
0925 09:13:40.980242 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:40.980246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:40.980265 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:13:40.980343 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:40.980346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:40.980417 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:40.980558 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:13:40.981166 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.981914 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:40.983307 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.983772 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:40.983923 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:40.984354 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:13:40.984995 (+ 641us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:13:40.985138 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:40.985212 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 09:13:40.985329 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":308,"threads_started":1}
I20260925 09:13:41.003199 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:40.996918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:34027 (local address 127.3.19.67:36073)
0925 09:13:40.997095 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:40.997099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:40.997731 (+ 632us) server_negotiation.cc:407] Connection header received
0925 09:13:40.997949 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:40.997954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:40.998025 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:40.998120 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:13:40.998625 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.999460 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:41.001430 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:41.001899 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:41.002040 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:41.002472 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:13:41.002572 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:13:41.002665 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:41.002818 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 09:13:41.002896 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 09:13:41.003304 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:40.996946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:36073 (local address 127.3.19.65:34027)
0925 09:13:40.997612 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:40.997628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:40.997806 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:40.998138 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:40.998142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:40.998154 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 09:13:40.998470 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:40.998478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:40.999592 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:40.999600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:41.001277 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:41.001295 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:41.002073 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:41.002079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:41.002222 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:41.002277 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 09:13:41.002668 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 09:13:41.002682 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 09:13:41.002779 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 09:13:41.002926 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":371,"threads_started":1}
I20260925 09:13:41.012609 5210 raft_consensus.cc:3037] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:41.012854 5349 raft_consensus.cc:1261] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Refusing update from remote peer 3d9fb7c7e27548b8a56c8a052dcf5dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:41.013644 5558 consensus_queue.cc:1035] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b326626ea524046967381848bde308b" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:41.016933 5210 raft_consensus.cc:1261] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Refusing update from remote peer 3d9fb7c7e27548b8a56c8a052dcf5dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:13:41.017845 5558 consensus_queue.cc:1035] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24e3ca0e87a44506a1d8cf084e98cc90" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:13:41.045698 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:41.039123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:59397 (local address 127.3.19.65:45725)
0925 09:13:41.039559 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:41.039567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:41.040007 (+ 440us) server_negotiation.cc:407] Connection header received
0925 09:13:41.040234 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:41.040240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:41.040322 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:41.040421 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:13:41.040920 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:41.041831 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:41.043523 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:41.044197 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:41.044387 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:41.044868 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:13:41.044980 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:13:41.045098 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:41.045284 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 09:13:41.045371 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318}
I20260925 09:13:41.045864 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:41.039114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:45725 (local address 127.3.19.67:59397)
0925 09:13:41.039894 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:41.039917 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:41.040062 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:41.040448 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:41.040453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:41.040468 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 09:13:41.040766 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:41.040775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:41.041970 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:41.041980 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:41.043359 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:41.043372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:41.044411 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:41.044418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:41.044600 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:41.044641 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 09:13:41.045099 (+ 458us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 09:13:41.045113 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 09:13:41.045233 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 09:13:41.045433 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":464,"threads_started":1}
W20260925 09:13:41.047219 5532 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 09:13:42.652451 4911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:13:42.652679 4911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:13:42.657192 4911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2298906d4cd4abb836d8f29b8013b0c: Sending DeleteTablet for 3 replicas of tablet 5648c0a24a8d44c38da37f0f4a78a7fc
I20260925 09:13:42.658100 5050 tablet_service.cc:1514] Processing DeleteTablet for tablet 5648c0a24a8d44c38da37f0f4a78a7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:42 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:51348
I20260925 09:13:42.658201 5329 tablet_service.cc:1514] Processing DeleteTablet for tablet 5648c0a24a8d44c38da37f0f4a78a7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:42 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:60296
I20260925 09:13:42.658305 5189 tablet_service.cc:1514] Processing DeleteTablet for tablet 5648c0a24a8d44c38da37f0f4a78a7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:13:42 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:36490
I20260925 09:13:42.658798 5592 tablet_replica.cc:330] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0: stopping tablet replica
I20260925 09:13:42.659170 5592 raft_consensus.cc:2229] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:13:42.660503 5592 raft_consensus.cc:2258] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:42.661724 5593 tablet_replica.cc:330] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b: stopping tablet replica
I20260925 09:13:42.661978 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 4983
I20260925 09:13:42.662092 5593 raft_consensus.cc:2229] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:13:42.662362 5593 raft_consensus.cc:2258] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:42.662458 5592 ts_tablet_manager.cc:1905] T 5648c0a24a8d44c38da37f0f4a78a7fc P 3d9fb7c7e27548b8a56c8a052dcf5dc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:42.663311 5594 tablet_replica.cc:330] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90: stopping tablet replica
I20260925 09:13:42.663583 5594 raft_consensus.cc:2229] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:13:42.663905 5594 raft_consensus.cc:2258] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:13:42.664906 5594 ts_tablet_manager.cc:1905] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:13:42.665318 5593 ts_tablet_manager.cc:1905] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 09:13:42 dist-test-slave-l6qh krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327619, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:13:42.690773 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:42.663368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.3.19.65:45725)
0925 09:13:42.668307 (+ 4939us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:42.668312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:42.668327 (+ 15us) server_negotiation.cc:407] Connection header received
0925 09:13:42.668374 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:42.668377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:42.668433 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:42.668575 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:42.669970 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:42.670805 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:42.672164 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:42.672656 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:42.672800 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:42.683747 (+ 10947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:42.683764 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:42.683788 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:42.683836 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:42.687204 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:42.688157 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:42.688167 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:42.688169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:42.688333 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:42.688874 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:42.688879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:42.688881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:42.689383 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:42.689741 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:42.690067 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 09:13:42.690321 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4833,"thread_start_us":4735,"threads_started":1}
W20260925 09:13:42.691606 5115 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54514
W20260925 09:13:42.692457 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 4983: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:13:42.692729 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 4983
I20260925 09:13:42.753549 5593 ts_tablet_manager.cc:1918] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 09:13:42.753549 5594 ts_tablet_manager.cc:1918] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 09:13:42.753734 5593 log.cc:1196] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/5648c0a24a8d44c38da37f0f4a78a7fc
I20260925 09:13:42.753734 5594 log.cc:1196] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/5648c0a24a8d44c38da37f0f4a78a7fc
I20260925 09:13:42.754717 5593 ts_tablet_manager.cc:1939] T 5648c0a24a8d44c38da37f0f4a78a7fc P 0b326626ea524046967381848bde308b: Deleting consensus metadata
I20260925 09:13:42.754716 5594 ts_tablet_manager.cc:1939] T 5648c0a24a8d44c38da37f0f4a78a7fc P 24e3ca0e87a44506a1d8cf084e98cc90: Deleting consensus metadata
W20260925 09:13:42.764531 4898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.19.65:45725 (error 108)
I20260925 09:13:42.765524 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 5124
I20260925 09:13:42.767148 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:42.766113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:45725 (local address 127.0.0.1:54524)
0925 09:13:42.766805 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:42.766934 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":435,"threads_started":1}
W20260925 09:13:42.767431 4898 catalog_manager.cc:4584] TS 3d9fb7c7e27548b8a56c8a052dcf5dc0 (127.3.19.65:45725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5648c0a24a8d44c38da37f0f4a78a7fc: Network error: Client connection negotiation failed: client connection to 127.3.19.65:45725: connect: Connection refused (error 111)
Sep 25 09:13:42 dist-test-slave-l6qh krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327619, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:13:42.782372 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:42.766297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.3.19.66:38795)
0925 09:13:42.767044 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:42.767048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:42.767065 (+ 17us) server_negotiation.cc:407] Connection header received
0925 09:13:42.767117 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:42.767121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:42.767181 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:42.767314 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:42.768631 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:42.769445 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:42.770690 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:42.771107 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:42.771246 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:42.775981 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:42.775996 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:42.776013 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:42.776050 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:42.779238 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:42.780027 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:42.780036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:42.780038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:42.780200 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:42.780632 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:42.780639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:42.780642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:42.781035 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:42.781373 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:42.781720 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 09:13:42.781976 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":520,"threads_started":1}
W20260925 09:13:42.783084 5255 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45790
W20260925 09:13:42.783773 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 5124: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:13:42.783928 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 5124
I20260925 09:13:42.821269 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:42.820766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:45725 (local address 127.0.0.1:54538)
0925 09:13:42.820963 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:42.821078 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 09:13:42.822400 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:42.822021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:45725 (local address 127.0.0.1:54552)
0925 09:13:42.822146 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:42.822228 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 09:13:42.825172 4899 catalog_manager.cc:4841] TS 0b326626ea524046967381848bde308b (127.3.19.67:36073): tablet 5648c0a24a8d44c38da37f0f4a78a7fc (table test-table [id=5409ebd7212b4a7d89035d80269bbe3c]) successfully deleted
I20260925 09:13:42.847676 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 5264
I20260925 09:13:42.848546 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:42.847872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:38795 (local address 127.0.0.1:45802)
0925 09:13:42.848225 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:42.848345 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.66:38795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
Sep 25 09:13:42 dist-test-slave-l6qh krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327619, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:13:42.865203 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:42.849079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.3.19.67:36073)
0925 09:13:42.849714 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:42.849719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:42.849734 (+ 15us) server_negotiation.cc:407] Connection header received
0925 09:13:42.849784 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:42.849788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:42.849848 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:42.849986 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:42.851358 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:42.852126 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:42.853497 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:42.853988 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:42.854126 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:42.859232 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:42.859249 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:42.859265 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:42.859312 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:42.862288 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:42.863058 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:42.863067 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:42.863069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:42.863207 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:42.863609 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:42.863614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:42.863619 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:42.863967 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:42.864265 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:42.864599 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 09:13:42.864828 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":372,"threads_started":1}
W20260925 09:13:42.866073 5395 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56586
W20260925 09:13:42.866659 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 5264: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:13:42.866853 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 5264
I20260925 09:13:42.881042 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:42.880527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:38795 (local address 127.0.0.1:45818)
0925 09:13:42.880744 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:42.880844 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.66:38795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 09:13:42.882334 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:42.881858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:38795 (local address 127.0.0.1:45832)
0925 09:13:42.882053 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:42.882147 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.66:38795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 09:13:42.882907 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:42.882192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:45725 (local address 127.0.0.1:54568)
0925 09:13:42.882574 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:42.882674 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"threads_started":1}
I20260925 09:13:42.884286 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:42.883692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:45725 (local address 127.0.0.1:54584)
0925 09:13:42.883895 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:42.884071 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 09:13:42.890173 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 4880
I20260925 09:13:42.891422 4952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:43.008118 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:43.007501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:38795 (local address 127.0.0.1:45848)
0925 09:13:43.007742 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:43.007858 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.66:38795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 09:13:43.008451 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:43.008036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:45725 (local address 127.0.0.1:54590)
0925 09:13:43.008210 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:43.008296 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 09:13:43.008435 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 4880
2026-09-25T09:13:43Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9615 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:13:44 dist-test-slave-l6qh krb5kdc[5605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:13:44 dist-test-slave-l6qh krb5kdc[5605](info): set up 2 sockets
Sep 25 09:13:44 dist-test-slave-l6qh krb5kdc[5605](info): commencing operation
krb5kdc: starting...
W20260925 09:13:45.258674 3149 mini_kdc.cc:121] Time spent starting KDC: real 2.203s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:13:45 dist-test-slave-l6qh krb5kdc[5605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:13:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:13:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:13:45.461807 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46343
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:46343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:45.816527 5618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:45.816908 5618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:45.817081 5618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:45.824404 5618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:13:45.824587 5618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:45.824700 5618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:45.824803 5618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:13:45.824898 5618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:13:45.835287 5618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:46343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:45.836746 5618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:45.838194 5618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:45.851188 5623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:45.851151 5626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:45.851119 5624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:45.851828 5618 server_base.cc:1034] running on GCE node
I20260925 09:13:45.852702 5618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:45.854300 5618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:45.855654 5618 hybrid_clock.cc:648] HybridClock initialized: now 1790327625855594 us; error 40 us; skew 500 ppm
I20260925 09:13:45.856017 5618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:45 dist-test-slave-l6qh krb5kdc[5605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:45.859195 5618 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:13:45.860810 5618 webserver.cc:460] Webserver started at http://127.3.19.126:43537/ using document root <none> and password file <none>
I20260925 09:13:45.861596 5618 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:45.861742 5618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:45.862064 5618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:45.865123 5618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "64d06ce4d4994b56b5a74011c67a8722"
format_stamp: "Formatted at 2026-09-25 09:13:45 on dist-test-slave-l6qh"
server_key: "260ff517f7da735c1347027b50948efb"
server_key_iv: "a59af4aa3b4d8576f63b7cc7059602f6"
server_key_version: "encryptionkey@0"
I20260925 09:13:45.866019 5618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "64d06ce4d4994b56b5a74011c67a8722"
format_stamp: "Formatted at 2026-09-25 09:13:45 on dist-test-slave-l6qh"
server_key: "260ff517f7da735c1347027b50948efb"
server_key_iv: "a59af4aa3b4d8576f63b7cc7059602f6"
server_key_version: "encryptionkey@0"
I20260925 09:13:45.871587 5618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:13:45.875413 5633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:45.876920 5618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:13:45.877123 5618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "64d06ce4d4994b56b5a74011c67a8722"
format_stamp: "Formatted at 2026-09-25 09:13:45 on dist-test-slave-l6qh"
server_key: "260ff517f7da735c1347027b50948efb"
server_key_iv: "a59af4aa3b4d8576f63b7cc7059602f6"
server_key_version: "encryptionkey@0"
I20260925 09:13:45.877472 5618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:45.908080 5618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:45.912859 5618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:45.913208 5618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:45.932197 5618 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:46343
I20260925 09:13:45.932235 5684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:46343 every 8 connection(s)
I20260925 09:13:45.933722 5618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:13:45.937879 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:45.942350 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 5618
I20260925 09:13:45.942615 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:13:45.943390 3149 external_mini_cluster.cc:1428] Setting key 0c25df3dddf05976396d28517abea4d1
I20260925 09:13:45.950660 5685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722: Bootstrap starting.
I20260925 09:13:45.954468 5685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:45.955766 5685 log.cc:824] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:45.958864 5685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722: No bootstrap required, opened a new log
Sep 25 09:13:45 dist-test-slave-l6qh krb5kdc[5605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:45.968281 5688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:45.945644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.3.19.126:46343)
0925 09:13:45.946480 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:45.946493 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:45.946537 (+ 44us) server_negotiation.cc:407] Connection header received
0925 09:13:45.948941 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:45.948968 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:45.949490 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:45.949836 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:45.951675 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:45.953020 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:45.954882 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:45.955634 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:45.955973 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:45.960615 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:45.960623 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:45.960646 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:45.960691 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:45.963871 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:45.964824 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:45.964833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:45.964841 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:45.965015 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:45.965462 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:45.965465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:45.965468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:45.966021 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:45.966254 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:45.966720 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 09:13:45.967161 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":385,"threads_started":1}
I20260925 09:13:45.968499 5685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d06ce4d4994b56b5a74011c67a8722" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46343 } }
I20260925 09:13:45.968950 5685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:45.969055 5685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d06ce4d4994b56b5a74011c67a8722, State: Initialized, Role: FOLLOWER
I20260925 09:13:45.969758 5685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d06ce4d4994b56b5a74011c67a8722" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46343 } }
I20260925 09:13:45.970019 5685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:13:45.970168 5685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:13:45.970362 5685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:45.972672 5685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d06ce4d4994b56b5a74011c67a8722" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46343 } }
I20260925 09:13:45.973199 5685 leader_election.cc:302] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64d06ce4d4994b56b5a74011c67a8722; no voters:
I20260925 09:13:45.973898 5685 leader_election.cc:288] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:13:45.974069 5690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:45.975256 5690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [term 1 LEADER]: Becoming Leader. State: Replica: 64d06ce4d4994b56b5a74011c67a8722, State: Running, Role: LEADER
I20260925 09:13:45.975806 5690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d06ce4d4994b56b5a74011c67a8722" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46343 } }
I20260925 09:13:45.976116 5685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:13:45.979139 5691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64d06ce4d4994b56b5a74011c67a8722" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d06ce4d4994b56b5a74011c67a8722" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46343 } } }
I20260925 09:13:45.979508 5691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:45.978986 5692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64d06ce4d4994b56b5a74011c67a8722. Latest consensus state: current_term: 1 leader_uuid: "64d06ce4d4994b56b5a74011c67a8722" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d06ce4d4994b56b5a74011c67a8722" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46343 } } }
I20260925 09:13:45.979748 5692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722 [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:45.982239 5697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:13:45.989209 5697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:13:45.997822 5697 catalog_manager.cc:1347] Generated new cluster ID: d2ab1c4498af4df6ba37e8fc1f40bc48
I20260925 09:13:45.997951 5697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:13:46.025695 5697 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:13:46.028359 5697 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:13:46.038223 5697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64d06ce4d4994b56b5a74011c67a8722: Generated new TSK 0
I20260925 09:13:46.039376 5697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:13:46.105046 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46343
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:46.440213 5711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:46.440573 5711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:46.440754 5711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:46.448189 5711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:46.448374 5711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:46.448552 5711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:13:46.458783 5711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:46.460160 5711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:46.461620 5711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:46.475411 5717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:46.475420 5716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:46.475945 5719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:46.476606 5711 server_base.cc:1034] running on GCE node
I20260925 09:13:46.477231 5711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:46.478279 5711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:46.479483 5711 hybrid_clock.cc:648] HybridClock initialized: now 1790327626479417 us; error 63 us; skew 500 ppm
I20260925 09:13:46.479807 5711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:46 dist-test-slave-l6qh krb5kdc[5605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327626, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:46.483354 5711 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:13:46.484911 5711 webserver.cc:460] Webserver started at http://127.3.19.65:33197/ using document root <none> and password file <none>
I20260925 09:13:46.485666 5711 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:46.485824 5711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:46.486160 5711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:46.489102 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "5332df24633d443090bc68d8720b5ac9"
format_stamp: "Formatted at 2026-09-25 09:13:46 on dist-test-slave-l6qh"
server_key: "41ae12a600e6ecf4283fd491e7746d48"
server_key_iv: "165009e150d05154651b0bb28bbaff90"
server_key_version: "encryptionkey@0"
I20260925 09:13:46.489976 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "5332df24633d443090bc68d8720b5ac9"
format_stamp: "Formatted at 2026-09-25 09:13:46 on dist-test-slave-l6qh"
server_key: "41ae12a600e6ecf4283fd491e7746d48"
server_key_iv: "165009e150d05154651b0bb28bbaff90"
server_key_version: "encryptionkey@0"
I20260925 09:13:46.496443 5711 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 09:13:46.500174 5726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:46.501701 5711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:13:46.501902 5711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "5332df24633d443090bc68d8720b5ac9"
format_stamp: "Formatted at 2026-09-25 09:13:46 on dist-test-slave-l6qh"
server_key: "41ae12a600e6ecf4283fd491e7746d48"
server_key_iv: "165009e150d05154651b0bb28bbaff90"
server_key_version: "encryptionkey@0"
I20260925 09:13:46.502115 5711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:46.517204 5711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:46.521535 5711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:46.521831 5711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:46.522815 5711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:46.524174 5711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:46.524281 5711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:46.524439 5711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:46.524516 5711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:46.558692 5711 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:44975
I20260925 09:13:46.558740 5838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:44975 every 8 connection(s)
I20260925 09:13:46.560081 5711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:13:46.567006 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 5711
I20260925 09:13:46.567245 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:13:46.567840 3149 external_mini_cluster.cc:1428] Setting key 6b84388c2accc6de0215febbcd5e4762
Sep 25 09:13:46 dist-test-slave-l6qh krb5kdc[5605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327626, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:46.583150 5841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:46.562754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:35241 (local address 127.3.19.126:46343)
0925 09:13:46.563269 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:46.563277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:46.564156 (+ 879us) server_negotiation.cc:407] Connection header received
0925 09:13:46.566230 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:46.566236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:46.566339 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:46.566472 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:46.568449 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:46.569486 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:46.571280 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:46.572094 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:46.572261 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:46.576206 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:46.576212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:46.576215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:46.576249 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:46.579374 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:46.580470 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:46.580475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:46.580477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:46.580639 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:46.581150 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:46.581153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:46.581155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:46.581546 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:46.581742 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:46.582660 (+ 918us) server_negotiation.cc:299] Negotiation successful
0925 09:13:46.582871 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":295,"threads_started":1}
I20260925 09:13:46.584764 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:46.563203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46343 (local address 127.3.19.65:35241)
0925 09:13:46.563969 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:46.564016 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:46.565920 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:46.566740 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:46.566751 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:46.567495 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:46.568226 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:46.568264 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:46.569675 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:46.569680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:46.571089 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:46.571103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:46.572340 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:46.572345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:46.572579 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:46.573360 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:46.573385 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:46.575925 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:46.579553 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:46.579567 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:46.579589 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:46.580314 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:46.580764 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:46.580772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:46.580778 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:46.581035 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:46.581684 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:46.581695 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:46.582200 (+ 505us) client_negotiation.cc:769] Sending connection context
0925 09:13:46.582562 (+ 362us) client_negotiation.cc:240] Negotiation successful
0925 09:13:46.583184 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":295,"threads_started":1}
I20260925 09:13:46.586621 5839 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46343
I20260925 09:13:46.587004 5839 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
I20260925 09:13:46.587857 5839 heartbeater.cc:511] Master 127.3.19.126:46343 requested a full tablet report, sending...
I20260925 09:13:46.590240 5650 ts_manager.cc:194] Registered new tserver with Master: 5332df24633d443090bc68d8720b5ac9 (127.3.19.65:44975)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
I20260925 09:13:46.591761 5650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:35241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:13:46.615944 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46343
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:46.952785 5845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:46.953086 5845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:46.953241 5845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:46.960675 5845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:46.960807 5845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:46.960922 5845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:13:46.970919 5845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:46.972214 5845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:46.973472 5845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:46.987123 5850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:46.987161 5851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:46.987056 5853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:46.988603 5845 server_base.cc:1034] running on GCE node
I20260925 09:13:46.989374 5845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:46.990480 5845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:46.991715 5845 hybrid_clock.cc:648] HybridClock initialized: now 1790327626991661 us; error 46 us; skew 500 ppm
I20260925 09:13:46.992010 5845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:46 dist-test-slave-l6qh krb5kdc[5605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327626, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:46.995280 5845 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:13:46.996870 5845 webserver.cc:460] Webserver started at http://127.3.19.66:42589/ using document root <none> and password file <none>
I20260925 09:13:46.997656 5845 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:46.997787 5845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:46.998116 5845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:47.001217 5845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "4f2853db20ed4fd287e4c8afe4fa474f"
format_stamp: "Formatted at 2026-09-25 09:13:46 on dist-test-slave-l6qh"
server_key: "cb90a2d862de335555ca0319c61f14a7"
server_key_iv: "cb74fad23d242f1f024e15c335d54fcb"
server_key_version: "encryptionkey@0"
I20260925 09:13:47.002111 5845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "4f2853db20ed4fd287e4c8afe4fa474f"
format_stamp: "Formatted at 2026-09-25 09:13:46 on dist-test-slave-l6qh"
server_key: "cb90a2d862de335555ca0319c61f14a7"
server_key_iv: "cb74fad23d242f1f024e15c335d54fcb"
server_key_version: "encryptionkey@0"
I20260925 09:13:47.007663 5845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:13:47.011170 5860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:47.012616 5845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:13:47.012810 5845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "4f2853db20ed4fd287e4c8afe4fa474f"
format_stamp: "Formatted at 2026-09-25 09:13:46 on dist-test-slave-l6qh"
server_key: "cb90a2d862de335555ca0319c61f14a7"
server_key_iv: "cb74fad23d242f1f024e15c335d54fcb"
server_key_version: "encryptionkey@0"
I20260925 09:13:47.013007 5845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:47.027529 5845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:47.032131 5845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:47.032481 5845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:47.033527 5845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:47.034952 5845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:47.035024 5845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:47.035167 5845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:47.035220 5845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:47.069379 5845 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:34375
I20260925 09:13:47.069434 5972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:34375 every 8 connection(s)
I20260925 09:13:47.070871 5845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:13:47.075801 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 5845
I20260925 09:13:47.076010 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:13:47.076434 3149 external_mini_cluster.cc:1428] Setting key e1ba88f248f4197f7fe02933ec353e8d
Sep 25 09:13:47 dist-test-slave-l6qh krb5kdc[5605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327626, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:47.094009 5841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:47.073421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:37081 (local address 127.3.19.126:46343)
0925 09:13:47.073598 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:47.073602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:47.074780 (+ 1178us) server_negotiation.cc:407] Connection header received
0925 09:13:47.076863 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:47.076896 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:47.076994 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:47.077113 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:47.078854 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:47.079576 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:47.082257 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:47.082748 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:47.082869 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:47.087395 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:47.087402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:47.087406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:47.087448 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:47.090391 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:47.091451 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:47.091455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:47.091457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:47.091583 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:47.092153 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:47.092158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:47.092162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:47.092450 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:47.092562 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:47.093595 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0925 09:13:47.093775 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 09:13:47.095729 5975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:47.073890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46343 (local address 127.3.19.66:37081)
0925 09:13:47.074610 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:47.074647 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:47.076568 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:47.077392 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:47.077401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:47.077929 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:47.078667 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:47.078695 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:47.079740 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:47.079749 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:47.082087 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:47.082105 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:47.082992 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:47.083001 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:47.083342 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:47.084117 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:47.084153 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:47.087058 (+ 2905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:47.090574 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:47.090587 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:47.090609 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:47.091298 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:47.091702 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:47.091710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:47.091716 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:47.091996 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:47.092589 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:47.092600 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:47.093120 (+ 520us) client_negotiation.cc:769] Sending connection context
0925 09:13:47.093510 (+ 390us) client_negotiation.cc:240] Negotiation successful
0925 09:13:47.094166 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":278,"threads_started":1}
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
I20260925 09:13:47.097378 5973 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46343
I20260925 09:13:47.097718 5973 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:47.098474 5973 heartbeater.cc:511] Master 127.3.19.126:46343 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
I20260925 09:13:47.100258 5650 ts_manager.cc:194] Registered new tserver with Master: 4f2853db20ed4fd287e4c8afe4fa474f (127.3.19.66:34375)
I20260925 09:13:47.101573 5650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:37081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:13:47.125484 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46343
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:47.451994 5978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:47.452368 5978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:47.452548 5978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:47.460103 5978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:47.460271 5978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:47.460460 5978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:13:47.470348 5978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:47.471724 5978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:47.473038 5978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:47.486654 5983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:47.486604 5984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:47.486603 5986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:47.487557 5978 server_base.cc:1034] running on GCE node
I20260925 09:13:47.488232 5978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:47.489208 5978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:47.490474 5978 hybrid_clock.cc:648] HybridClock initialized: now 1790327627490420 us; error 49 us; skew 500 ppm
I20260925 09:13:47.490804 5978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:47 dist-test-slave-l6qh krb5kdc[5605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:47.494181 5978 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:13:47.496635 5978 webserver.cc:460] Webserver started at http://127.3.19.67:41319/ using document root <none> and password file <none>
I20260925 09:13:47.497537 5978 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:47.497680 5978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:47.498018 5978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:47.500990 5978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "db8b955b359143a2a1a9eeb252f05480"
format_stamp: "Formatted at 2026-09-25 09:13:47 on dist-test-slave-l6qh"
server_key: "871ea7889cfc0c9325191326eca5617a"
server_key_iv: "256e9b180e8e9a8351b2c7fb716ec951"
server_key_version: "encryptionkey@0"
I20260925 09:13:47.501914 5978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "db8b955b359143a2a1a9eeb252f05480"
format_stamp: "Formatted at 2026-09-25 09:13:47 on dist-test-slave-l6qh"
server_key: "871ea7889cfc0c9325191326eca5617a"
server_key_iv: "256e9b180e8e9a8351b2c7fb716ec951"
server_key_version: "encryptionkey@0"
I20260925 09:13:47.507483 5978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:13:47.511000 5993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:47.512488 5978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:13:47.512683 5978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "db8b955b359143a2a1a9eeb252f05480"
format_stamp: "Formatted at 2026-09-25 09:13:47 on dist-test-slave-l6qh"
server_key: "871ea7889cfc0c9325191326eca5617a"
server_key_iv: "256e9b180e8e9a8351b2c7fb716ec951"
server_key_version: "encryptionkey@0"
I20260925 09:13:47.512918 5978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:47.540756 5978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:47.545188 5978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:47.545553 5978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:47.546602 5978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:47.548015 5978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:47.548125 5978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:47.548278 5978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:47.548350 5978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:47.583971 5978 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:36093
I20260925 09:13:47.584020 6105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:36093 every 8 connection(s)
I20260925 09:13:47.585354 5978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:13:47.594777 5839 heartbeater.cc:503] Master 127.3.19.126:46343 was elected leader, sending a full tablet report...
I20260925 09:13:47.595594 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 5978
I20260925 09:13:47.595836 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:13:47.596378 3149 external_mini_cluster.cc:1428] Setting key ad348da2b6d626b90f33390cc68f4b50
Sep 25 09:13:47 dist-test-slave-l6qh krb5kdc[5605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:47.607244 5841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:47.588067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:60859 (local address 127.3.19.126:46343)
0925 09:13:47.588272 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:47.588279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:47.589686 (+ 1407us) server_negotiation.cc:407] Connection header received
0925 09:13:47.591538 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:47.591542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:47.591644 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:47.591740 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:47.593377 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:47.594231 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:47.596229 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:47.596908 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:47.597082 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:47.601351 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:47.601358 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:47.601363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:47.601408 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:47.604055 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:47.604838 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:47.604845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:47.604849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:47.604985 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:47.605516 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:47.605523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:47.605526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:47.605809 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:47.605961 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:47.606720 (+ 759us) server_negotiation.cc:299] Negotiation successful
0925 09:13:47.606927 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 09:13:47.608287 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:47.588677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46343 (local address 127.3.19.67:60859)
0925 09:13:47.589503 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:47.589552 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:47.591239 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:47.591959 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:47.591969 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:47.592541 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:47.593114 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:47.593132 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:47.594387 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:47.594391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:47.596069 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:47.596084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:47.597185 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:47.597195 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:47.597578 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:47.598400 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:47.598425 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:47.601098 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:47.604224 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:47.604234 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:47.604254 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:47.604706 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:47.605103 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:47.605108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:47.605112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:47.605400 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:47.605937 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:47.605945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:47.606322 (+ 377us) client_negotiation.cc:769] Sending connection context
0925 09:13:47.606604 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 09:13:47.607090 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":303,"threads_started":1}
I20260925 09:13:47.610044 6106 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46343
I20260925 09:13:47.610432 6106 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:47.611269 6106 heartbeater.cc:511] Master 127.3.19.126:46343 requested a full tablet report, sending...
I20260925 09:13:47.612855 5650 ts_manager.cc:194] Registered new tserver with Master: db8b955b359143a2a1a9eeb252f05480 (127.3.19.67:36093)
I20260925 09:13:47.613780 5650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:60859
I20260925 09:13:47.624290 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 09:13:47.659094 5841 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:13:47.656300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.3.19.126:46343)
0925 09:13:47.656512 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:47.656519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:47.656991 (+ 472us) server_negotiation.cc:407] Connection header received
0925 09:13:47.657174 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:47.657180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:47.657323 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:47.657458 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:47.658884 (+ 1426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40206: BlockingRecv error: recv got EOF from 127.0.0.1:40206 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 09:13:47.662556 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 5711
W20260925 09:13:47.665916 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 5711: Not authorized: Client connection negotiation failed: client connection to 127.3.19.65:44975: server requires authentication, but client does not have Kerberos credentials available
I20260925 09:13:47.666100 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 5711
I20260925 09:13:47.682561 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 5845
W20260925 09:13:47.685186 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 5845: Not authorized: Client connection negotiation failed: client connection to 127.3.19.66:34375: server requires authentication, but client does not have Kerberos credentials available
I20260925 09:13:47.685412 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 5845
I20260925 09:13:47.701417 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 5978
W20260925 09:13:47.704708 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 5978: Not authorized: Client connection negotiation failed: client connection to 127.3.19.67:36093: server requires authentication, but client does not have Kerberos credentials available
I20260925 09:13:47.704843 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 5978
I20260925 09:13:47.720808 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 5618
I20260925 09:13:47.722078 5680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:47.824116 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 5618
2026-09-25T09:13:47Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4811 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:13:47 dist-test-slave-l6qh krb5kdc[6123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:13:47 dist-test-slave-l6qh krb5kdc[6123](info): set up 2 sockets
Sep 25 09:13:47 dist-test-slave-l6qh krb5kdc[6123](info): commencing operation
krb5kdc: starting...
W20260925 09:13:48.939064 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:13:49 dist-test-slave-l6qh krb5kdc[6123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:13:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:13:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:13:49.144990 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:39821
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:39821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:49.475706 6136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:49.476018 6136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:49.476186 6136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:49.483671 6136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:13:49.483781 6136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:49.483891 6136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:49.484018 6136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:13:49.484104 6136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:13:49.494233 6136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:39821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:39821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:49.495610 6136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:49.497020 6136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:49.510317 6144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:49.510303 6141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:49.510576 6142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:49.511121 6136 server_base.cc:1034] running on GCE node
I20260925 09:13:49.511987 6136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:49.513513 6136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:49.514746 6136 hybrid_clock.cc:648] HybridClock initialized: now 1790327629514689 us; error 57 us; skew 500 ppm
I20260925 09:13:49.515120 6136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:49 dist-test-slave-l6qh krb5kdc[6123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327629, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:49.518420 6136 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:13:49.520015 6136 webserver.cc:460] Webserver started at http://127.3.19.126:43999/ using document root <none> and password file <none>
I20260925 09:13:49.520735 6136 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:49.520888 6136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:49.521207 6136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:49.524403 6136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "d0916c462aeb42a3844ee3c4894d27d0"
format_stamp: "Formatted at 2026-09-25 09:13:49 on dist-test-slave-l6qh"
server_key: "a53f7a7125d9cec5edc7f7737ed214fb"
server_key_iv: "70e41e47728814b1359d17ad90027035"
server_key_version: "encryptionkey@0"
I20260925 09:13:49.525300 6136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "d0916c462aeb42a3844ee3c4894d27d0"
format_stamp: "Formatted at 2026-09-25 09:13:49 on dist-test-slave-l6qh"
server_key: "a53f7a7125d9cec5edc7f7737ed214fb"
server_key_iv: "70e41e47728814b1359d17ad90027035"
server_key_version: "encryptionkey@0"
I20260925 09:13:49.530865 6136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 09:13:49.535001 6151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:49.536468 6136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:13:49.536666 6136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "d0916c462aeb42a3844ee3c4894d27d0"
format_stamp: "Formatted at 2026-09-25 09:13:49 on dist-test-slave-l6qh"
server_key: "a53f7a7125d9cec5edc7f7737ed214fb"
server_key_iv: "70e41e47728814b1359d17ad90027035"
server_key_version: "encryptionkey@0"
I20260925 09:13:49.536885 6136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:49.572409 6136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:49.577111 6136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:49.577447 6136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:49.597174 6136 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:39821
I20260925 09:13:49.597220 6202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:39821 every 8 connection(s)
I20260925 09:13:49.598690 6136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:13:49.602555 6203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:49.606050 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 6136
I20260925 09:13:49.606267 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:13:49.607062 3149 external_mini_cluster.cc:1428] Setting key 8f15505b0ff3e4efc7eddd5954f83ed1
I20260925 09:13:49.616340 6203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0: Bootstrap starting.
I20260925 09:13:49.620398 6203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:49.621881 6203 log.cc:824] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0: Log is configured to *not* fsync() on all Append() calls
Sep 25 09:13:49 dist-test-slave-l6qh krb5kdc[6123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:49.625372 6203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0: No bootstrap required, opened a new log
I20260925 09:13:49.631362 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:49.609181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.3.19.126:39821)
0925 09:13:49.610207 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:49.610220 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:49.610267 (+ 47us) server_negotiation.cc:407] Connection header received
0925 09:13:49.611989 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:49.612012 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:49.612475 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:49.612856 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:49.614722 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:49.616058 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:49.617898 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:49.618686 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:49.618849 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:49.623836 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:49.623845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:49.623861 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:49.623904 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:49.627169 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:49.628021 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:49.628028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:49.628034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:49.628208 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:49.628593 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:49.628596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:49.628598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:49.629148 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:49.629475 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:49.629844 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 09:13:49.630237 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":450,"threads_started":1}
I20260925 09:13:49.634356 6203 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0916c462aeb42a3844ee3c4894d27d0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39821 } }
I20260925 09:13:49.634780 6203 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:49.634925 6203 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0916c462aeb42a3844ee3c4894d27d0, State: Initialized, Role: FOLLOWER
I20260925 09:13:49.635471 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:13:49.635576 6203 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0916c462aeb42a3844ee3c4894d27d0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39821 } }
I20260925 09:13:49.635825 6203 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:13:49.635955 6203 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:13:49.636097 6203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:49.639551 6203 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0916c462aeb42a3844ee3c4894d27d0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39821 } }
I20260925 09:13:49.640170 6203 leader_election.cc:302] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0916c462aeb42a3844ee3c4894d27d0; no voters:
I20260925 09:13:49.640890 6203 leader_election.cc:288] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:13:49.641024 6208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:49.643431 6203 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:13:49.644202 6208 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [term 1 LEADER]: Becoming Leader. State: Replica: d0916c462aeb42a3844ee3c4894d27d0, State: Running, Role: LEADER
I20260925 09:13:49.644771 6208 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0916c462aeb42a3844ee3c4894d27d0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39821 } }
I20260925 09:13:49.648185 6210 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0916c462aeb42a3844ee3c4894d27d0. Latest consensus state: current_term: 1 leader_uuid: "d0916c462aeb42a3844ee3c4894d27d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0916c462aeb42a3844ee3c4894d27d0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39821 } } }
I20260925 09:13:49.653971 6210 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:49.648097 6209 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0916c462aeb42a3844ee3c4894d27d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0916c462aeb42a3844ee3c4894d27d0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39821 } } }
I20260925 09:13:49.654439 6209 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0 [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:49.656198 6219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:13:49.664933 6219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:13:49.670483 6227 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 09:13:49.675518 6219 catalog_manager.cc:1347] Generated new cluster ID: 238d82421ffa4c1a9405df7a1f5d8c12
I20260925 09:13:49.675639 6219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:13:49.697022 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:49.687753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.3.19.126:39821)
0925 09:13:49.687949 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:49.687957 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:49.688616 (+ 659us) server_negotiation.cc:407] Connection header received
0925 09:13:49.688803 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:49.688815 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:49.688893 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:49.688995 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:49.690297 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:49.691373 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:49.693261 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:49.693906 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:49.694018 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:49.695305 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:49.695312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:49.695317 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:13:49.696171 (+ 854us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:49.696249 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:13:49.696282 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:49.696394 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:49.696589 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 09:13:49.696683 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 09:13:49.702368 6219 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:13:49.703936 6219 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:13:49.720232 6219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0916c462aeb42a3844ee3c4894d27d0: Generated new TSK 0
I20260925 09:13:49.721578 6219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:13:49.731556 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 6136
I20260925 09:13:49.732738 6198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:49.833590 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 6136
2026-09-25T09:13:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2007 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:13:49 dist-test-slave-l6qh krb5kdc[6237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:13:49 dist-test-slave-l6qh krb5kdc[6237](info): set up 2 sockets
Sep 25 09:13:49 dist-test-slave-l6qh krb5kdc[6237](info): commencing operation
krb5kdc: starting...
W20260925 09:13:50.990613 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.005s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:13:51 dist-test-slave-l6qh krb5kdc[6237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:13:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:13:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:13:51.197074 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:37261
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:37261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:51.533535 6250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:51.533921 6250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:51.534077 6250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:51.541803 6250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:13:51.541972 6250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:51.542080 6250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:51.542188 6250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:13:51.542264 6250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:13:51.552714 6250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:37261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:37261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:51.554207 6250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:51.555546 6250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:51.568147 6256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:51.568220 6255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:51.568248 6258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:51.569095 6250 server_base.cc:1034] running on GCE node
I20260925 09:13:51.570119 6250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:51.571967 6250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:51.573279 6250 hybrid_clock.cc:648] HybridClock initialized: now 1790327631573171 us; error 57 us; skew 500 ppm
I20260925 09:13:51.573710 6250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:51 dist-test-slave-l6qh krb5kdc[6237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:51.577500 6250 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:13:51.579177 6250 webserver.cc:460] Webserver started at http://127.3.19.126:34477/ using document root <none> and password file <none>
I20260925 09:13:51.579892 6250 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:51.580008 6250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:51.580276 6250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:51.583482 6250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "cc5ad3dd9ab64ddebc2aca85d1c4eafd"
format_stamp: "Formatted at 2026-09-25 09:13:51 on dist-test-slave-l6qh"
server_key: "ab97316f462a997725085fbd81045ba8"
server_key_iv: "8b92c67c3e76d48aed629789e2a11af3"
server_key_version: "encryptionkey@0"
I20260925 09:13:51.584285 6250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "cc5ad3dd9ab64ddebc2aca85d1c4eafd"
format_stamp: "Formatted at 2026-09-25 09:13:51 on dist-test-slave-l6qh"
server_key: "ab97316f462a997725085fbd81045ba8"
server_key_iv: "8b92c67c3e76d48aed629789e2a11af3"
server_key_version: "encryptionkey@0"
I20260925 09:13:51.589972 6250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 09:13:51.593708 6265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:51.595189 6250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 09:13:51.595340 6250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "cc5ad3dd9ab64ddebc2aca85d1c4eafd"
format_stamp: "Formatted at 2026-09-25 09:13:51 on dist-test-slave-l6qh"
server_key: "ab97316f462a997725085fbd81045ba8"
server_key_iv: "8b92c67c3e76d48aed629789e2a11af3"
server_key_version: "encryptionkey@0"
I20260925 09:13:51.595520 6250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:51.613924 6250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:51.618235 6250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:51.618484 6250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:51.637015 6250 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:37261
I20260925 09:13:51.637058 6316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:37261 every 8 connection(s)
I20260925 09:13:51.638445 6250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:13:51.642352 6317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:51.647903 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 6250
I20260925 09:13:51.648131 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:13:51.648936 3149 external_mini_cluster.cc:1428] Setting key 81bd1b456c00b35d0f227597ab2e7182
I20260925 09:13:51.655124 6317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd: Bootstrap starting.
I20260925 09:13:51.659325 6317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:51.660544 6317 log.cc:824] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:51.664145 6317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd: No bootstrap required, opened a new log
Sep 25 09:13:51 dist-test-slave-l6qh krb5kdc[6237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:51.674144 6317 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5ad3dd9ab64ddebc2aca85d1c4eafd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37261 } }
I20260925 09:13:51.674676 6317 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:51.674777 6317 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5ad3dd9ab64ddebc2aca85d1c4eafd, State: Initialized, Role: FOLLOWER
I20260925 09:13:51.675472 6317 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5ad3dd9ab64ddebc2aca85d1c4eafd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37261 } }
I20260925 09:13:51.675734 6317 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:13:51.675890 6317 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:13:51.676077 6317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:51.678956 6317 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5ad3dd9ab64ddebc2aca85d1c4eafd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37261 } }
I20260925 09:13:51.679694 6317 leader_election.cc:302] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc5ad3dd9ab64ddebc2aca85d1c4eafd; no voters:
I20260925 09:13:51.680392 6317 leader_election.cc:288] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:13:51.680581 6322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:51.681854 6322 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [term 1 LEADER]: Becoming Leader. State: Replica: cc5ad3dd9ab64ddebc2aca85d1c4eafd, State: Running, Role: LEADER
I20260925 09:13:51.682397 6322 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5ad3dd9ab64ddebc2aca85d1c4eafd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37261 } }
I20260925 09:13:51.682878 6317 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:13:51.683516 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:51.652546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.3.19.126:37261)
0925 09:13:51.659739 (+ 7193us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:51.659771 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:51.659817 (+ 46us) server_negotiation.cc:407] Connection header received
0925 09:13:51.662163 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:51.662188 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:51.662597 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:51.662949 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:51.664771 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:51.666176 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:51.668044 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:51.668878 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:51.669120 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:51.673940 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:51.673951 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:51.673971 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:51.674024 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:51.677925 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:51.678806 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:51.678816 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:51.678825 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:51.679047 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:51.679554 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:51.679558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:51.679560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:51.680338 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:51.680717 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:51.681389 (+ 672us) server_negotiation.cc:299] Negotiation successful
0925 09:13:51.681888 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6898,"thread_start_us":6667,"threads_started":1}
I20260925 09:13:51.685322 6323 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc5ad3dd9ab64ddebc2aca85d1c4eafd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5ad3dd9ab64ddebc2aca85d1c4eafd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37261 } } }
I20260925 09:13:51.685702 6323 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:51.689109 6324 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc5ad3dd9ab64ddebc2aca85d1c4eafd. Latest consensus state: current_term: 1 leader_uuid: "cc5ad3dd9ab64ddebc2aca85d1c4eafd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5ad3dd9ab64ddebc2aca85d1c4eafd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37261 } } }
I20260925 09:13:51.689589 6324 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:51.690845 6327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:13:51.697031 6327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:13:51.706931 6327 catalog_manager.cc:1347] Generated new cluster ID: 4ada2946c7d54107bf8ce5a5e2a908bf
I20260925 09:13:51.707104 6327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:13:51.731503 6327 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:13:51.732839 6327 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:13:51.754884 6327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc5ad3dd9ab64ddebc2aca85d1c4eafd: Generated new TSK 0
I20260925 09:13:51.755807 6327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:13:51.811252 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:37261
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:52.169677 6343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:52.170020 6343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:52.170248 6343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:52.177871 6343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:52.177984 6343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:52.178193 6343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:13:52.188912 6343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:37261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:52.190552 6343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:52.192034 6343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:52.206197 6351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:52.206272 6349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:52.206297 6348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:52.207186 6343 server_base.cc:1034] running on GCE node
I20260925 09:13:52.207913 6343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:52.208957 6343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:52.210207 6343 hybrid_clock.cc:648] HybridClock initialized: now 1790327632210132 us; error 62 us; skew 500 ppm
I20260925 09:13:52.210562 6343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:52 dist-test-slave-l6qh krb5kdc[6237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:52.214358 6343 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:13:52.215934 6343 webserver.cc:460] Webserver started at http://127.3.19.65:43241/ using document root <none> and password file <none>
I20260925 09:13:52.216672 6343 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:52.216825 6343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:52.217142 6343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:52.220117 6343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "97ef296a825a4de9842a2605a7ee0ab4"
format_stamp: "Formatted at 2026-09-25 09:13:52 on dist-test-slave-l6qh"
server_key: "040e6c5c456f4d414d5d4d5cd172e0cc"
server_key_iv: "1b7380a6dcafe83540815757d833126e"
server_key_version: "encryptionkey@0"
I20260925 09:13:52.221000 6343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "97ef296a825a4de9842a2605a7ee0ab4"
format_stamp: "Formatted at 2026-09-25 09:13:52 on dist-test-slave-l6qh"
server_key: "040e6c5c456f4d414d5d4d5cd172e0cc"
server_key_iv: "1b7380a6dcafe83540815757d833126e"
server_key_version: "encryptionkey@0"
I20260925 09:13:52.226416 6343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 09:13:52.229915 6358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:52.231359 6343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:13:52.231550 6343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "97ef296a825a4de9842a2605a7ee0ab4"
format_stamp: "Formatted at 2026-09-25 09:13:52 on dist-test-slave-l6qh"
server_key: "040e6c5c456f4d414d5d4d5cd172e0cc"
server_key_iv: "1b7380a6dcafe83540815757d833126e"
server_key_version: "encryptionkey@0"
I20260925 09:13:52.231763 6343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:52.247120 6343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:52.251484 6343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:52.251806 6343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:52.252777 6343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:52.254335 6343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:52.254436 6343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:52.254580 6343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:52.254657 6343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:52.289325 6343 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:45583
I20260925 09:13:52.289414 6470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:45583 every 8 connection(s)
I20260925 09:13:52.290663 6343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:13:52.294560 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 6343
I20260925 09:13:52.294772 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:13:52.295508 3149 external_mini_cluster.cc:1428] Setting key 2e2446766f45676b67776776fb58cae6
Sep 25 09:13:52 dist-test-slave-l6qh krb5kdc[6237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:52.312059 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:52.293462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:60259 (local address 127.3.19.126:37261)
0925 09:13:52.293908 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:52.293913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:52.295017 (+ 1104us) server_negotiation.cc:407] Connection header received
0925 09:13:52.297086 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:52.297090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:52.297164 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:52.297430 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:52.299217 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:52.300051 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:52.301743 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:52.302143 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:52.302287 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:52.306135 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:52.306140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:52.306143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:52.306175 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:52.308767 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:52.309699 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:52.309706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:52.309715 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:52.309835 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:52.310283 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:52.310293 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:52.310295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:52.310560 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:52.310717 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:52.311448 (+ 731us) server_negotiation.cc:299] Negotiation successful
0925 09:13:52.311680 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":246,"threads_started":1}
I20260925 09:13:52.313078 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:52.293921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:37261 (local address 127.3.19.65:60259)
0925 09:13:52.294812 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:52.294889 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:52.296804 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:52.297716 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:52.297756 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:52.298352 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:52.299017 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:52.299051 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:52.300215 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:52.300225 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:52.301584 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:52.301595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:52.302397 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:52.302406 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:52.302648 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:52.303426 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:52.303448 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:52.305909 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:52.308996 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:52.309004 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:52.309019 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:52.309565 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:52.309958 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:52.309963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:52.309968 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:52.310160 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:52.310691 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:52.310699 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:52.311071 (+ 372us) client_negotiation.cc:769] Sending connection context
0925 09:13:52.311339 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 09:13:52.311916 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":371,"threads_started":1}
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
I20260925 09:13:52.314814 6471 heartbeater.cc:347] Connected to a master server at 127.3.19.126:37261
I20260925 09:13:52.315151 6471 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:52.315964 6471 heartbeater.cc:511] Master 127.3.19.126:37261 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
I20260925 09:13:52.318483 6282 ts_manager.cc:194] Registered new tserver with Master: 97ef296a825a4de9842a2605a7ee0ab4 (127.3.19.65:45583)
I20260925 09:13:52.319974 6282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:60259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:13:52.341104 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:37261
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:52.672798 6477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:52.673094 6477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:52.673187 6477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:52.680612 6477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:52.680725 6477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:52.680840 6477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:13:52.691180 6477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:37261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:52.692549 6477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:52.693849 6477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:52.707719 6483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:52.707789 6482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:52.707813 6485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:52.708583 6477 server_base.cc:1034] running on GCE node
I20260925 09:13:52.709199 6477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:52.710209 6477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:52.711470 6477 hybrid_clock.cc:648] HybridClock initialized: now 1790327632711424 us; error 93 us; skew 500 ppm
I20260925 09:13:52.711818 6477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:52 dist-test-slave-l6qh krb5kdc[6237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:52.715065 6477 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:13:52.716596 6477 webserver.cc:460] Webserver started at http://127.3.19.66:37003/ using document root <none> and password file <none>
I20260925 09:13:52.717325 6477 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:52.717427 6477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:52.717681 6477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:52.720597 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "6ebf5b144c5441b49ae9ee88ae1637b6"
format_stamp: "Formatted at 2026-09-25 09:13:52 on dist-test-slave-l6qh"
server_key: "699de48a1d25a0e126786487a7e14c00"
server_key_iv: "f7a96b682a5614d512ef0051bd19ef2d"
server_key_version: "encryptionkey@0"
I20260925 09:13:52.721424 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "6ebf5b144c5441b49ae9ee88ae1637b6"
format_stamp: "Formatted at 2026-09-25 09:13:52 on dist-test-slave-l6qh"
server_key: "699de48a1d25a0e126786487a7e14c00"
server_key_iv: "f7a96b682a5614d512ef0051bd19ef2d"
server_key_version: "encryptionkey@0"
I20260925 09:13:52.726640 6477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 09:13:52.730257 6492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:52.731727 6477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 09:13:52.731922 6477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "6ebf5b144c5441b49ae9ee88ae1637b6"
format_stamp: "Formatted at 2026-09-25 09:13:52 on dist-test-slave-l6qh"
server_key: "699de48a1d25a0e126786487a7e14c00"
server_key_iv: "f7a96b682a5614d512ef0051bd19ef2d"
server_key_version: "encryptionkey@0"
I20260925 09:13:52.732131 6477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:52.760842 6477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:52.766054 6477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:52.766423 6477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:52.767575 6477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:52.769179 6477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:52.769345 6477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 09:13:52.769505 6477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:52.769606 6477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:52.807421 6477 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:43507
I20260925 09:13:52.807461 6604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:43507 every 8 connection(s)
I20260925 09:13:52.808907 6477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:13:52.811311 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 6477
I20260925 09:13:52.811532 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:13:52.812033 3149 external_mini_cluster.cc:1428] Setting key 43b7cea0370f8acb0c524ead8dcb662a
Sep 25 09:13:52 dist-test-slave-l6qh krb5kdc[6237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
I20260925 09:13:52.831664 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:52.811796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:50139 (local address 127.3.19.126:37261)
0925 09:13:52.812040 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:52.812049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:52.813499 (+ 1450us) server_negotiation.cc:407] Connection header received
0925 09:13:52.815518 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:52.815529 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:52.815637 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:52.815802 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:52.817325 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:52.818671 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:52.820432 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:52.820909 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:52.821025 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:52.825597 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:52.825602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:52.825606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:52.825645 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:52.828493 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:52.829325 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:52.829332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:52.829335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:52.829471 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:52.829916 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:52.829921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:52.829924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:52.830202 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:52.830370 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:52.831077 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 09:13:52.831308 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 09:13:52.832743 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:52.812452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:37261 (local address 127.3.19.66:50139)
0925 09:13:52.813314 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:52.813369 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:52.815230 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:52.815972 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:52.815981 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:52.816529 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:52.817112 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:52.817129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:52.818843 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:52.818847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:52.820262 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:52.820287 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:52.821161 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:52.821170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:52.821543 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:52.822321 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:52.822355 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:52.825310 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:52.828674 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:52.828684 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:52.828704 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:52.829159 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:52.829588 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:52.829593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:52.829597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:52.829790 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:52.830330 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:52.830339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:52.830699 (+ 360us) client_negotiation.cc:769] Sending connection context
0925 09:13:52.830977 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 09:13:52.831533 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":343,"threads_started":1}
I20260925 09:13:52.834304 6605 heartbeater.cc:347] Connected to a master server at 127.3.19.126:37261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
I20260925 09:13:52.834647 6605 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:52.835604 6605 heartbeater.cc:511] Master 127.3.19.126:37261 requested a full tablet report, sending...
I20260925 09:13:52.837424 6282 ts_manager.cc:194] Registered new tserver with Master: 6ebf5b144c5441b49ae9ee88ae1637b6 (127.3.19.66:43507)
I20260925 09:13:52.838677 6282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:50139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:13:52.860483 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:37261
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:53.186519 6610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:53.186811 6610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:53.186908 6610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:53.194412 6610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:53.194528 6610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:53.194638 6610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:13:53.204643 6610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:37261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:53.205986 6610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:53.207216 6610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:53.220443 6618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:53.220484 6616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:53.220520 6615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:53.221248 6610 server_base.cc:1034] running on GCE node
I20260925 09:13:53.221925 6610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:53.222929 6610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:53.224165 6610 hybrid_clock.cc:648] HybridClock initialized: now 1790327633224128 us; error 64 us; skew 500 ppm
I20260925 09:13:53.224500 6610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:53 dist-test-slave-l6qh krb5kdc[6237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327633, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:53.235347 6610 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:13:53.237066 6610 webserver.cc:460] Webserver started at http://127.3.19.67:34527/ using document root <none> and password file <none>
I20260925 09:13:53.237847 6610 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:53.237991 6610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:53.238298 6610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:53.241307 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "33069b70fbb443d9b575f5df4a9da5cc"
format_stamp: "Formatted at 2026-09-25 09:13:53 on dist-test-slave-l6qh"
server_key: "7c0e79cb6668ccf43b8b820f96de8c5c"
server_key_iv: "dd4904058af57b26fec97c292dcd76ca"
server_key_version: "encryptionkey@0"
I20260925 09:13:53.242226 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "33069b70fbb443d9b575f5df4a9da5cc"
format_stamp: "Formatted at 2026-09-25 09:13:53 on dist-test-slave-l6qh"
server_key: "7c0e79cb6668ccf43b8b820f96de8c5c"
server_key_iv: "dd4904058af57b26fec97c292dcd76ca"
server_key_version: "encryptionkey@0"
I20260925 09:13:53.247342 6610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:13:53.250954 6625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:53.252346 6610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:13:53.252554 6610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "33069b70fbb443d9b575f5df4a9da5cc"
format_stamp: "Formatted at 2026-09-25 09:13:53 on dist-test-slave-l6qh"
server_key: "7c0e79cb6668ccf43b8b820f96de8c5c"
server_key_iv: "dd4904058af57b26fec97c292dcd76ca"
server_key_version: "encryptionkey@0"
I20260925 09:13:53.252879 6610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:53.281131 6610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:53.285864 6610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:53.286198 6610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:53.287227 6610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:53.288805 6610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:53.288898 6610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:53.289008 6610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:53.289068 6610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:53.323035 6610 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:43359
I20260925 09:13:53.323084 6737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:43359 every 8 connection(s)
I20260925 09:13:53.323933 6471 heartbeater.cc:503] Master 127.3.19.126:37261 was elected leader, sending a full tablet report...
I20260925 09:13:53.324429 6610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:13:53.330724 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 6610
I20260925 09:13:53.330926 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:13:53.331359 3149 external_mini_cluster.cc:1428] Setting key 562453e14c42e6de11a1a825bcf4a676
Sep 25 09:13:53 dist-test-slave-l6qh krb5kdc[6237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327633, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:53.348196 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:53.327456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:37261 (local address 127.3.19.67:38327)
0925 09:13:53.328119 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:53.328160 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:53.329794 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:53.330403 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:53.330411 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:53.331023 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:53.331657 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:53.331676 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:53.332954 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:53.332959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:53.334270 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:53.334281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:53.335071 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:53.335078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:53.335300 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:53.336012 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:53.336035 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:53.339155 (+ 3120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:53.343376 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:53.343387 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:53.343432 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:53.344136 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:53.344679 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:53.344683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:53.344686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:53.344934 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:53.345680 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:53.345691 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:53.346150 (+ 459us) client_negotiation.cc:769] Sending connection context
0925 09:13:53.346458 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 09:13:53.346995 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":254,"threads_started":1}
I20260925 09:13:53.349390 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:53.327050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:38327 (local address 127.3.19.126:37261)
0925 09:13:53.327222 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:53.327227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:53.328275 (+ 1048us) server_negotiation.cc:407] Connection header received
0925 09:13:53.330026 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:53.330031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:53.330096 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:53.330182 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:53.331826 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:53.332819 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:53.334410 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:53.334833 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:53.334960 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:53.339511 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:53.339521 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:53.339526 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:53.339585 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:53.343219 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:53.344279 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:53.344285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:53.344289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:53.344508 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:53.345064 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:53.345069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:53.345072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:53.345462 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:53.345612 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:53.348780 (+ 3168us) server_negotiation.cc:299] Negotiation successful
0925 09:13:53.349015 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 09:13:53.350636 6738 heartbeater.cc:347] Connected to a master server at 127.3.19.126:37261
I20260925 09:13:53.350979 6738 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:53.351723 6738 heartbeater.cc:511] Master 127.3.19.126:37261 requested a full tablet report, sending...
I20260925 09:13:53.353432 6282 ts_manager.cc:194] Registered new tserver with Master: 33069b70fbb443d9b575f5df4a9da5cc (127.3.19.67:43359)
I20260925 09:13:53.354322 6282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:38327
I20260925 09:13:53.359105 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 09:13:53 dist-test-slave-l6qh krb5kdc[6237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327633, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 09:13:53 dist-test-slave-l6qh krb5kdc[6237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327633, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:53.410440 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:53.395497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.3.19.126:37261)
0925 09:13:53.395735 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:53.395744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:53.396120 (+ 376us) server_negotiation.cc:407] Connection header received
0925 09:13:53.396371 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:53.396378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:53.396458 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:53.396579 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:53.398103 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:53.398949 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:53.400303 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:53.400796 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:53.400943 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:53.405155 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:53.405160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:53.405164 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:53.405197 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:53.407905 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:53.408564 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:53.408568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:53.408571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:53.408701 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:53.409124 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:53.409128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:53.409130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:53.409580 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:53.409783 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:53.410018 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 09:13:53.410178 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 09:13:53.411031 6282 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52016
I20260925 09:13:53.412983 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 6343
Sep 25 09:13:53 dist-test-slave-l6qh krb5kdc[6237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327633, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:13:53.429352 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:53.414088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.3.19.65:45583)
0925 09:13:53.414736 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:53.414742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:53.415114 (+ 372us) server_negotiation.cc:407] Connection header received
0925 09:13:53.415214 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:53.415220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:53.415453 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:53.415584 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:53.416670 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:53.417718 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:53.418858 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:53.419423 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:53.419520 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:53.423477 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:53.423490 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:53.423505 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:53.423545 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:53.426483 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:53.427177 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:53.427193 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:53.427195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:53.427324 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:53.427707 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:53.427713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:53.427717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:53.428065 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:53.428380 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:53.428651 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 09:13:53.428925 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":324,"threads_started":1}
W20260925 09:13:53.430649 6466 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43702
W20260925 09:13:53.431824 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 6343: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:13:53.432060 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 6343
I20260925 09:13:53.448202 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 6477
Sep 25 09:13:53 dist-test-slave-l6qh krb5kdc[6237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327633, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:13:53.465111 6750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:53.449083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.3.19.66:43507)
0925 09:13:53.449642 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:53.449650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:53.449676 (+ 26us) server_negotiation.cc:407] Connection header received
0925 09:13:53.449736 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:53.449741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:53.449960 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:53.450137 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:53.451417 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:53.452428 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:53.453795 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:53.454391 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:53.454560 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:53.459112 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:53.459124 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:53.459139 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:53.459176 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:53.462123 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:53.462818 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:53.462826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:53.462828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:53.462973 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:53.463349 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:53.463360 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:53.463363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:53.463715 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:53.464050 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:53.464357 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 09:13:53.464638 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":318,"threads_started":1}
W20260925 09:13:53.466502 6600 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41942
W20260925 09:13:53.467712 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 6477: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:13:53.467861 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 6477
I20260925 09:13:53.483423 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 6610
Sep 25 09:13:53 dist-test-slave-l6qh krb5kdc[6237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327633, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:13:53.501332 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:53.484457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.3.19.67:43359)
0925 09:13:53.485094 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:53.485101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:53.485131 (+ 30us) server_negotiation.cc:407] Connection header received
0925 09:13:53.485191 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:53.485197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:53.485515 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:53.485699 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:53.487061 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:53.488111 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:53.489408 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:53.489987 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:53.490132 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:53.494669 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:53.494682 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:53.494697 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:53.494732 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:53.498084 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:53.498868 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:53.498879 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:53.498884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:53.499071 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:53.499483 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:53.499491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:53.499494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:53.499874 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:53.500207 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:53.500539 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 09:13:53.500826 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":334,"threads_started":1}
W20260925 09:13:53.502702 6733 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49392
W20260925 09:13:53.503862 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 6610: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:13:53.504024 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 6610
I20260925 09:13:53.519641 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 6250
I20260925 09:13:53.521166 6312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:53.623625 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 6250
2026-09-25T09:13:53Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3793 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:13:53 dist-test-slave-l6qh krb5kdc[6755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:13:53 dist-test-slave-l6qh krb5kdc[6755](info): set up 2 sockets
Sep 25 09:13:53 dist-test-slave-l6qh krb5kdc[6755](info): commencing operation
krb5kdc: starting...
W20260925 09:13:54.714717 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:13:54 dist-test-slave-l6qh krb5kdc[6755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:13:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:13:54.925192 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:36795
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:36795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:55.257980 6768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:55.258270 6768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:55.258373 6768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:55.265748 6768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:13:55.265868 6768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:55.265914 6768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:55.265971 6768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:13:55.266018 6768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:13:55.276100 6768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:36795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:36795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:55.277582 6768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:55.278930 6768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:55.291513 6773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:55.291569 6774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:55.291546 6776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:55.292400 6768 server_base.cc:1034] running on GCE node
I20260925 09:13:55.293323 6768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:55.294955 6768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:55.296216 6768 hybrid_clock.cc:648] HybridClock initialized: now 1790327635296162 us; error 54 us; skew 500 ppm
I20260925 09:13:55.296628 6768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:55 dist-test-slave-l6qh krb5kdc[6755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:55.300014 6768 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:13:55.301702 6768 webserver.cc:460] Webserver started at http://127.3.19.126:39839/ using document root <none> and password file <none>
I20260925 09:13:55.302423 6768 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:55.302528 6768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:55.302796 6768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:55.305964 6768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "211dd2144e6745b8be18676b55bed7a9"
format_stamp: "Formatted at 2026-09-25 09:13:55 on dist-test-slave-l6qh"
server_key: "d3178c90dde66eb20ff9a1236c6ea783"
server_key_iv: "ffa11d858628dbcdaaa7a9ab9590b57a"
server_key_version: "encryptionkey@0"
I20260925 09:13:55.306756 6768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "211dd2144e6745b8be18676b55bed7a9"
format_stamp: "Formatted at 2026-09-25 09:13:55 on dist-test-slave-l6qh"
server_key: "d3178c90dde66eb20ff9a1236c6ea783"
server_key_iv: "ffa11d858628dbcdaaa7a9ab9590b57a"
server_key_version: "encryptionkey@0"
I20260925 09:13:55.312148 6768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:13:55.315929 6783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:55.317611 6768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:13:55.317776 6768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "211dd2144e6745b8be18676b55bed7a9"
format_stamp: "Formatted at 2026-09-25 09:13:55 on dist-test-slave-l6qh"
server_key: "d3178c90dde66eb20ff9a1236c6ea783"
server_key_iv: "ffa11d858628dbcdaaa7a9ab9590b57a"
server_key_version: "encryptionkey@0"
I20260925 09:13:55.317963 6768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:55.339390 6768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:55.343799 6768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:55.344064 6768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:55.363044 6768 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:36795
I20260925 09:13:55.363066 6834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:36795 every 8 connection(s)
I20260925 09:13:55.364488 6768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:13:55.365761 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 6768
I20260925 09:13:55.366007 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:13:55.366808 3149 external_mini_cluster.cc:1428] Setting key f93da6baf7cc449825d38b0946448da9
I20260925 09:13:55.369597 6835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 09:13:55 dist-test-slave-l6qh krb5kdc[6755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:55.383281 6835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9: Bootstrap starting.
I20260925 09:13:55.386580 6835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:55.387637 6835 log.cc:824] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:55.390601 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:55.368886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.3.19.126:36795)
0925 09:13:55.369769 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:55.369781 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:55.369821 (+ 40us) server_negotiation.cc:407] Connection header received
0925 09:13:55.371440 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:55.371464 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:55.371848 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:55.372212 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:55.374056 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:55.375454 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:55.377318 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:55.378174 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:55.378315 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:55.383071 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:55.383079 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:55.383096 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:55.383141 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:55.386303 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:55.387126 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:55.387134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:55.387141 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:55.387310 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:55.387701 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:55.387707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:55.387709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:55.388476 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:55.388801 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:55.389155 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 09:13:55.389635 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":386,"threads_started":1}
I20260925 09:13:55.390929 6835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9: No bootstrap required, opened a new log
I20260925 09:13:55.398267 6835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211dd2144e6745b8be18676b55bed7a9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36795 } }
I20260925 09:13:55.398658 6835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:55.398770 6835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 211dd2144e6745b8be18676b55bed7a9, State: Initialized, Role: FOLLOWER
I20260925 09:13:55.399439 6835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211dd2144e6745b8be18676b55bed7a9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36795 } }
I20260925 09:13:55.399732 6835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:13:55.399870 6835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:13:55.400043 6835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:55.402261 6835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211dd2144e6745b8be18676b55bed7a9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36795 } }
I20260925 09:13:55.402782 6835 leader_election.cc:302] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 211dd2144e6745b8be18676b55bed7a9; no voters:
I20260925 09:13:55.403481 6835 leader_election.cc:288] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:13:55.403674 6840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:55.405516 6840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [term 1 LEADER]: Becoming Leader. State: Replica: 211dd2144e6745b8be18676b55bed7a9, State: Running, Role: LEADER
I20260925 09:13:55.405812 6835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:13:55.406137 6840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211dd2144e6745b8be18676b55bed7a9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36795 } }
I20260925 09:13:55.409408 6841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "211dd2144e6745b8be18676b55bed7a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211dd2144e6745b8be18676b55bed7a9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36795 } } }
I20260925 09:13:55.409464 6842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 211dd2144e6745b8be18676b55bed7a9. Latest consensus state: current_term: 1 leader_uuid: "211dd2144e6745b8be18676b55bed7a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211dd2144e6745b8be18676b55bed7a9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36795 } } }
I20260925 09:13:55.409857 6841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:55.409866 6842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9 [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:55.410777 6850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:13:55.415812 6850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:13:55.425037 6850 catalog_manager.cc:1347] Generated new cluster ID: 66c20ee48b924142998436a3df50c1fa
I20260925 09:13:55.425164 6850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:13:55.456352 6850 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:13:55.458714 6850 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:13:55.467878 6850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 211dd2144e6745b8be18676b55bed7a9: Generated new TSK 0
I20260925 09:13:55.468782 6850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:13:55.529881 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:36795
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:55.874229 6861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:55.874543 6861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:55.874709 6861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:55.882295 6861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:55.882426 6861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:55.882623 6861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:13:55.892895 6861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:36795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:55.894327 6861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:55.895699 6861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:55.909461 6867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:55.909713 6869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:55.910166 6866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:55.910949 6861 server_base.cc:1034] running on GCE node
I20260925 09:13:55.911619 6861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:55.912576 6861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:55.913770 6861 hybrid_clock.cc:648] HybridClock initialized: now 1790327635913703 us; error 68 us; skew 500 ppm
I20260925 09:13:55.914108 6861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:55 dist-test-slave-l6qh krb5kdc[6755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:55.917379 6861 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:13:55.918828 6861 webserver.cc:460] Webserver started at http://127.3.19.65:36145/ using document root <none> and password file <none>
I20260925 09:13:55.919548 6861 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:55.919697 6861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:55.920014 6861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:55.923122 6861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "f5dc0502be494876b6615e84cd9816e4"
format_stamp: "Formatted at 2026-09-25 09:13:55 on dist-test-slave-l6qh"
server_key: "a35d5138546ee8170d124e3187aef2ce"
server_key_iv: "cb13bec731782e9b5b260bb500897f70"
server_key_version: "encryptionkey@0"
I20260925 09:13:55.923889 6861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "f5dc0502be494876b6615e84cd9816e4"
format_stamp: "Formatted at 2026-09-25 09:13:55 on dist-test-slave-l6qh"
server_key: "a35d5138546ee8170d124e3187aef2ce"
server_key_iv: "cb13bec731782e9b5b260bb500897f70"
server_key_version: "encryptionkey@0"
I20260925 09:13:55.929132 6861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 09:13:55.932637 6876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:55.934149 6861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:13:55.934334 6861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "f5dc0502be494876b6615e84cd9816e4"
format_stamp: "Formatted at 2026-09-25 09:13:55 on dist-test-slave-l6qh"
server_key: "a35d5138546ee8170d124e3187aef2ce"
server_key_iv: "cb13bec731782e9b5b260bb500897f70"
server_key_version: "encryptionkey@0"
I20260925 09:13:55.934551 6861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:55.964422 6861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:55.969417 6861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:55.969769 6861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:55.970839 6861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:55.972397 6861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:55.972481 6861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:55.972653 6861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:55.972757 6861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:56.006987 6861 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:35797
I20260925 09:13:56.007022 6988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:35797 every 8 connection(s)
I20260925 09:13:56.008337 6861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:13:56.012830 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 6861
I20260925 09:13:56.013085 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:13:56.013926 3149 external_mini_cluster.cc:1428] Setting key 89777b127e44c23d2738641bad84d8e4
Sep 25 09:13:56 dist-test-slave-l6qh krb5kdc[6755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:56.032977 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:56.011030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:53583 (local address 127.3.19.126:36795)
0925 09:13:56.011487 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:56.011493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:56.012647 (+ 1154us) server_negotiation.cc:407] Connection header received
0925 09:13:56.015296 (+ 2649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:56.015302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:56.015394 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:56.015530 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:56.017851 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:56.018565 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:56.021294 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:56.021781 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:56.021898 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:56.025733 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:56.025742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:56.025750 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:56.025797 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:56.029403 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:56.030138 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:56.030146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:56.030153 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:56.030305 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:56.030830 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:56.030836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:56.030842 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:56.031278 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:56.031450 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:56.032541 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0925 09:13:56.032719 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":205,"threads_started":1}
I20260925 09:13:56.034745 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:56.011676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:36795 (local address 127.3.19.65:53583)
0925 09:13:56.012437 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:56.012502 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:56.014972 (+ 2470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:56.015795 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:56.015804 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:56.016706 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:56.017632 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:56.017661 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:56.018730 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:56.018738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:56.021035 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:56.021053 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:56.022052 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:56.022062 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:56.022412 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:56.023166 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:56.023191 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:56.025490 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:56.029555 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:56.029563 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:56.029577 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:56.029998 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:56.030468 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:56.030472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:56.030475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:56.030691 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:56.031446 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:56.031452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:56.032003 (+ 551us) client_negotiation.cc:769] Sending connection context
0925 09:13:56.032437 (+ 434us) client_negotiation.cc:240] Negotiation successful
0925 09:13:56.033146 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":298,"threads_started":1}
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
I20260925 09:13:56.036628 6989 heartbeater.cc:347] Connected to a master server at 127.3.19.126:36795
I20260925 09:13:56.037034 6989 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:56.038004 6989 heartbeater.cc:511] Master 127.3.19.126:36795 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
I20260925 09:13:56.040743 6800 ts_manager.cc:194] Registered new tserver with Master: f5dc0502be494876b6615e84cd9816e4 (127.3.19.65:35797)
I20260925 09:13:56.042994 6800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:53583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:13:56.065709 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:36795
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:56.391547 6995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:56.391839 6995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:56.391935 6995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:56.399295 6995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:56.399425 6995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:56.399539 6995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:13:56.409628 6995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:36795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:56.410934 6995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:56.412242 6995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:56.425977 7000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:56.426010 7003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:56.426036 7001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:56.427148 6995 server_base.cc:1034] running on GCE node
I20260925 09:13:56.427831 6995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:56.428766 6995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:56.429992 6995 hybrid_clock.cc:648] HybridClock initialized: now 1790327636429930 us; error 60 us; skew 500 ppm
I20260925 09:13:56.430318 6995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:56 dist-test-slave-l6qh krb5kdc[6755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:56.433758 6995 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:13:56.435428 6995 webserver.cc:460] Webserver started at http://127.3.19.66:33127/ using document root <none> and password file <none>
I20260925 09:13:56.436173 6995 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:56.436333 6995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:56.436669 6995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:56.439886 6995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "bff3ad0ee0014b3e82d04ec5bf22818b"
format_stamp: "Formatted at 2026-09-25 09:13:56 on dist-test-slave-l6qh"
server_key: "6b5d3efc4779f962c73a6d2bd87ff808"
server_key_iv: "17f393c5cb8b6e49c3b3eaee5083aca5"
server_key_version: "encryptionkey@0"
I20260925 09:13:56.440698 6995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "bff3ad0ee0014b3e82d04ec5bf22818b"
format_stamp: "Formatted at 2026-09-25 09:13:56 on dist-test-slave-l6qh"
server_key: "6b5d3efc4779f962c73a6d2bd87ff808"
server_key_iv: "17f393c5cb8b6e49c3b3eaee5083aca5"
server_key_version: "encryptionkey@0"
I20260925 09:13:56.446070 6995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 09:13:56.449965 7010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:56.451469 6995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:13:56.451681 6995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "bff3ad0ee0014b3e82d04ec5bf22818b"
format_stamp: "Formatted at 2026-09-25 09:13:56 on dist-test-slave-l6qh"
server_key: "6b5d3efc4779f962c73a6d2bd87ff808"
server_key_iv: "17f393c5cb8b6e49c3b3eaee5083aca5"
server_key_version: "encryptionkey@0"
I20260925 09:13:56.451915 6995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:56.466434 6995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:56.470963 6995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:56.471274 6995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:56.472235 6995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:56.473680 6995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:56.473785 6995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:56.473935 6995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:56.474014 6995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:56.507555 6995 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:33767
I20260925 09:13:56.507647 7122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:33767 every 8 connection(s)
I20260925 09:13:56.508914 6995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:13:56.515790 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 6995
I20260925 09:13:56.516012 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:13:56.516433 3149 external_mini_cluster.cc:1428] Setting key 417714d66d53d348ed104701f255d222
Sep 25 09:13:56 dist-test-slave-l6qh krb5kdc[6755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:56.530385 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:56.511733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:47379 (local address 127.3.19.126:36795)
0925 09:13:56.511948 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:56.511952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:56.513200 (+ 1248us) server_negotiation.cc:407] Connection header received
0925 09:13:56.515125 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:56.515131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:56.515228 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:56.515340 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:56.517070 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:56.517916 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:56.519704 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:56.520523 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:56.520672 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:56.524574 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:56.524579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:56.524582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:56.524615 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:56.527266 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:56.528028 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:56.528034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:56.528038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:56.528172 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:56.528615 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:56.528621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:56.528625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:56.528890 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:56.529062 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:56.529815 (+ 753us) server_negotiation.cc:299] Negotiation successful
0925 09:13:56.530052 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 09:13:56.531844 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:56.512253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:36795 (local address 127.3.19.66:47379)
0925 09:13:56.513015 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:56.513062 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:56.514829 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:56.515613 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:56.515621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:56.516255 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:56.516898 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:56.516915 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:56.518132 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:56.518137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:56.519537 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:56.519551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:56.520788 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:56.520793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:56.521015 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:56.521816 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:56.521852 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:56.524314 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:56.527440 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:56.527449 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:56.527481 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:56.527894 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:56.528295 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:56.528300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:56.528304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:56.528500 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:56.529024 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:56.529034 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:56.529458 (+ 424us) client_negotiation.cc:769] Sending connection context
0925 09:13:56.529734 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 09:13:56.530283 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":309,"threads_started":1}
I20260925 09:13:56.533816 7123 heartbeater.cc:347] Connected to a master server at 127.3.19.126:36795
I20260925 09:13:56.534163 7123 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:56.534917 7123 heartbeater.cc:511] Master 127.3.19.126:36795 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
I20260925 09:13:56.536543 6800 ts_manager.cc:194] Registered new tserver with Master: bff3ad0ee0014b3e82d04ec5bf22818b (127.3.19.66:33767)
I20260925 09:13:56.537495 6800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:47379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:13:56.564067 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:36795
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:56.899473 7128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:56.899770 7128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:56.899925 7128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:56.907557 7128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:56.907675 7128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:56.907838 7128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:13:56.917771 7128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:36795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:56.919122 7128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:56.920466 7128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:56.934237 7133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:56.934276 7136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:56.934216 7134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:56.935278 7128 server_base.cc:1034] running on GCE node
I20260925 09:13:56.935968 7128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:56.936957 7128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:56.938154 7128 hybrid_clock.cc:648] HybridClock initialized: now 1790327636938088 us; error 71 us; skew 500 ppm
I20260925 09:13:56.938498 7128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:56 dist-test-slave-l6qh krb5kdc[6755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:56.941701 7128 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:13:56.943311 7128 webserver.cc:460] Webserver started at http://127.3.19.67:32945/ using document root <none> and password file <none>
I20260925 09:13:56.944044 7128 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:56.944191 7128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:56.944507 7128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:56.947533 7128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "da3a8a63f5814af0bbb860a20e71b898"
format_stamp: "Formatted at 2026-09-25 09:13:56 on dist-test-slave-l6qh"
server_key: "d37a90e6301e9cb82e1a447e1f917bf8"
server_key_iv: "5b6a97881f504169f6814505e3f94e61"
server_key_version: "encryptionkey@0"
I20260925 09:13:56.948333 7128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "da3a8a63f5814af0bbb860a20e71b898"
format_stamp: "Formatted at 2026-09-25 09:13:56 on dist-test-slave-l6qh"
server_key: "d37a90e6301e9cb82e1a447e1f917bf8"
server_key_iv: "5b6a97881f504169f6814505e3f94e61"
server_key_version: "encryptionkey@0"
I20260925 09:13:56.953575 7128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 09:13:56.957208 7143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:56.958745 7128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:13:56.958931 7128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "da3a8a63f5814af0bbb860a20e71b898"
format_stamp: "Formatted at 2026-09-25 09:13:56 on dist-test-slave-l6qh"
server_key: "d37a90e6301e9cb82e1a447e1f917bf8"
server_key_iv: "5b6a97881f504169f6814505e3f94e61"
server_key_version: "encryptionkey@0"
I20260925 09:13:56.959175 7128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:56.984261 7128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:56.988481 7128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:56.988796 7128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:56.989831 7128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:56.991200 7128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:56.991305 7128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:56.991454 7128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:56.991525 7128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:57.026530 7128 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:40563
I20260925 09:13:57.026623 7255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:40563 every 8 connection(s)
I20260925 09:13:57.027791 7128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:13:57.035254 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 7128
I20260925 09:13:57.035486 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:13:57.036008 3149 external_mini_cluster.cc:1428] Setting key f950bacc1a34b69204306e5435bb51d2
Sep 25 09:13:57 dist-test-slave-l6qh krb5kdc[6755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:57.047029 6989 heartbeater.cc:503] Master 127.3.19.126:36795 was elected leader, sending a full tablet report...
I20260925 09:13:57.049587 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:57.030580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:58963 (local address 127.3.19.126:36795)
0925 09:13:57.030788 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:57.030793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:57.031956 (+ 1163us) server_negotiation.cc:407] Connection header received
0925 09:13:57.033822 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:57.033827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:57.033930 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:57.034024 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:57.035873 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.036715 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.038945 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.039476 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.039577 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:57.043094 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:57.043100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:57.043104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:57.043144 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:57.046072 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:57.046935 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:57.046941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:57.046944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:57.047133 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:57.047609 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:57.047614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:57.047624 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:57.047970 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:57.048114 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:57.048972 (+ 858us) server_negotiation.cc:299] Negotiation successful
0925 09:13:57.049162 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 09:13:57.050603 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:57.031055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:36795 (local address 127.3.19.67:58963)
0925 09:13:57.031770 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:57.031812 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:57.033559 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:57.034234 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:57.034241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:57.034756 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:57.035672 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:57.035700 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.036882 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.036888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:57.038731 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:57.038772 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.039675 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.039679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:57.039887 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:57.040531 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:57.040554 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:57.042837 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:13:57.046232 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:57.046240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:57.046254 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:57.046761 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:57.047281 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:13:57.047284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:13:57.047287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:13:57.047477 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:13:57.048105 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:13:57.048111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:13:57.048538 (+ 427us) client_negotiation.cc:769] Sending connection context
0925 09:13:57.048815 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 09:13:57.049221 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":297,"threads_started":1}
I20260925 09:13:57.052110 7256 heartbeater.cc:347] Connected to a master server at 127.3.19.126:36795
I20260925 09:13:57.052459 7256 heartbeater.cc:464] Registering TS with master...
I20260925 09:13:57.053166 7256 heartbeater.cc:511] Master 127.3.19.126:36795 requested a full tablet report, sending...
I20260925 09:13:57.054582 6800 ts_manager.cc:194] Registered new tserver with Master: da3a8a63f5814af0bbb860a20e71b898 (127.3.19.67:40563)
I20260925 09:13:57.055404 6800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:58963
I20260925 09:13:57.064368 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 09:13:57 dist-test-slave-l6qh krb5kdc[6755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 09:13:57 dist-test-slave-l6qh krb5kdc[6755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:13:57.115370 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:57.099586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.3.19.65:35797)
0925 09:13:57.100397 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:57.100411 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:57.100438 (+ 27us) server_negotiation.cc:407] Connection header received
0925 09:13:57.100497 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:57.100503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:57.100732 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:57.100864 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:57.102369 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.103396 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.104715 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.105567 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.105670 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:57.109768 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:57.109781 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:57.109797 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:57.109839 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:57.112766 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:57.113521 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:57.113529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:57.113531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:57.113668 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:57.114034 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:57.114037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:57.114039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:57.114371 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:57.114608 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:57.114886 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 09:13:57.115086 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":496,"threads_started":1}
Sep 25 09:13:57 dist-test-slave-l6qh krb5kdc[6755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:57.133090 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:57.119715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.3.19.126:36795)
0925 09:13:57.119935 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:57.119948 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:57.120041 (+ 93us) server_negotiation.cc:407] Connection header received
0925 09:13:57.120167 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:57.120171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:57.120233 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:57.120368 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:57.121651 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.122421 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.123664 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.124072 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.124162 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:57.128003 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:57.128009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:57.128013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:57.128055 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:57.130760 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:57.131387 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:57.131391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:57.131394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:57.131541 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:57.131844 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:57.131847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:57.131849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:57.132156 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:57.132335 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:57.132594 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 09:13:57.132755 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260925 09:13:57.133682 6800 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56824
I20260925 09:13:57.134711 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 6861
I20260925 09:13:57.147732 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:57.135681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.3.19.65:35797)
0925 09:13:57.135891 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:57.135897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:57.136096 (+ 199us) server_negotiation.cc:407] Connection header received
0925 09:13:57.136287 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:57.136292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:57.136383 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:57.136510 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:57.137773 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.138449 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.139669 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.140104 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.140241 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:57.142555 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:57.142560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:57.142563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:57.142596 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:57.145096 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:57.145825 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:57.145833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:57.145838 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:57.145973 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:57.146346 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:57.146351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:57.146354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:57.146612 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:57.146885 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:57.147201 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 09:13:57.147416 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 09:13:57.148370 6984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:57.231918 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 6861
I20260925 09:13:57.247984 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 6995
Sep 25 09:13:57 dist-test-slave-l6qh krb5kdc[6755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:13:57.264415 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:57.248899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.3.19.66:33767)
0925 09:13:57.249504 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:57.249511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:57.249537 (+ 26us) server_negotiation.cc:407] Connection header received
0925 09:13:57.249596 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:57.249603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:57.249808 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:57.249986 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:57.251228 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.252213 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.253620 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.254165 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.254303 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:57.258597 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:57.258607 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:57.258621 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:57.258653 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:57.261489 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:57.262179 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:57.262186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:57.262188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:57.262358 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:57.262743 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:57.262748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:57.262751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:57.263143 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:57.263464 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:57.263781 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 09:13:57.264049 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":263,"threads_started":1}
I20260925 09:13:57.265713 7118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:57.348968 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 6995
I20260925 09:13:57.366021 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 7128
Sep 25 09:13:57 dist-test-slave-l6qh krb5kdc[6755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:13:57.383942 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:57.367087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40700 (local address 127.3.19.67:40563)
0925 09:13:57.367829 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:57.367841 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:57.367878 (+ 37us) server_negotiation.cc:407] Connection header received
0925 09:13:57.367977 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:57.367986 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:57.368259 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:57.368418 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:57.369838 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.371157 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.372499 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:57.373090 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:57.373205 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:57.377685 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:57.377698 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:57.377713 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:57.377751 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:57.380787 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:57.381862 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:57.381873 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:57.381877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:57.382024 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:57.382493 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:57.382497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:57.382499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:57.382879 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:57.383137 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:57.383421 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 09:13:57.383632 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":404,"threads_started":1}
I20260925 09:13:57.385320 7251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:57.467935 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 7128
I20260925 09:13:57.484958 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 6768
I20260925 09:13:57.486284 6830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:13:57.589687 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 6768
2026-09-25T09:13:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3964 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:13:57 dist-test-slave-l6qh krb5kdc[7275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:13:57 dist-test-slave-l6qh krb5kdc[7275](info): set up 2 sockets
Sep 25 09:13:57 dist-test-slave-l6qh krb5kdc[7275](info): commencing operation
krb5kdc: starting...
W20260925 09:13:58.686797 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:13:58 dist-test-slave-l6qh krb5kdc[7275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:13:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:13:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:13:58.897339 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:44243
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:44243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:59.228237 7288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:59.228561 7288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:59.228655 7288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:59.236102 7288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:13:59.236225 7288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:59.236276 7288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:59.236333 7288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:13:59.236384 7288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:13:59.246691 7288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:44243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:44243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:59.247974 7288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:59.249269 7288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:59.261013 7293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:59.261057 7296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:59.261103 7294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:59.262019 7288 server_base.cc:1034] running on GCE node
I20260925 09:13:59.262898 7288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:59.264400 7288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:59.265656 7288 hybrid_clock.cc:648] HybridClock initialized: now 1790327639265588 us; error 64 us; skew 500 ppm
I20260925 09:13:59.266099 7288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:59 dist-test-slave-l6qh krb5kdc[7275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327639, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:59.275053 7288 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:13:59.276340 7288 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:59.276515 7288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:59.276851 7288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:59.280046 7288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "82253e063b544eed90b7f2a68076c2e9"
format_stamp: "Formatted at 2026-09-25 09:13:59 on dist-test-slave-l6qh"
server_key: "83e5726e9dba895f99b3a44c65f03cb8"
server_key_iv: "f6e742cefe0a044ae38094b8dc8555cf"
server_key_version: "encryptionkey@0"
I20260925 09:13:59.280877 7288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "82253e063b544eed90b7f2a68076c2e9"
format_stamp: "Formatted at 2026-09-25 09:13:59 on dist-test-slave-l6qh"
server_key: "83e5726e9dba895f99b3a44c65f03cb8"
server_key_iv: "f6e742cefe0a044ae38094b8dc8555cf"
server_key_version: "encryptionkey@0"
I20260925 09:13:59.286320 7288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 09:13:59.290217 7302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:59.291946 7288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:13:59.292177 7288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "82253e063b544eed90b7f2a68076c2e9"
format_stamp: "Formatted at 2026-09-25 09:13:59 on dist-test-slave-l6qh"
server_key: "83e5726e9dba895f99b3a44c65f03cb8"
server_key_iv: "f6e742cefe0a044ae38094b8dc8555cf"
server_key_version: "encryptionkey@0"
I20260925 09:13:59.292446 7288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:59.308660 7288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:59.313072 7288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:59.313402 7288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:59.332252 7288 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:44243
I20260925 09:13:59.332281 7353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:44243 every 8 connection(s)
I20260925 09:13:59.333192 7288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:13:59.337021 7354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:13:59.337623 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 7288
I20260925 09:13:59.337853 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:13:59.338481 3149 external_mini_cluster.cc:1428] Setting key a9cf5844b790a375b3998e664fda1692
I20260925 09:13:59.351704 7354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9: Bootstrap starting.
Sep 25 09:13:59 dist-test-slave-l6qh krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:13:59.354882 7354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9: Neither blocks nor log segments found. Creating new log.
I20260925 09:13:59.356220 7354 log.cc:824] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9: Log is configured to *not* fsync() on all Append() calls
I20260925 09:13:59.359704 7354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9: No bootstrap required, opened a new log
I20260925 09:13:59.363636 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:59.340570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.3.19.126:44243)
0925 09:13:59.341465 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:59.341482 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:59.341532 (+ 50us) server_negotiation.cc:407] Connection header received
0925 09:13:59.343809 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:59.343839 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:59.344460 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:59.344946 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:59.346467 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:59.348499 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:59.349964 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:59.350999 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:59.351190 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:59.355025 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:59.355036 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:59.355055 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:13:59.355107 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:13:59.358951 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:59.359644 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:59.359653 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:59.359664 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:59.359864 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:13:59.360246 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:13:59.360251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:13:59.360254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:13:59.361013 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:13:59.361320 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 09:13:59.361729 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 09:13:59.362187 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":366,"threads_started":1}
I20260925 09:13:59.369108 7354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82253e063b544eed90b7f2a68076c2e9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44243 } }
I20260925 09:13:59.369514 7354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:13:59.369591 7354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82253e063b544eed90b7f2a68076c2e9, State: Initialized, Role: FOLLOWER
I20260925 09:13:59.370142 7354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82253e063b544eed90b7f2a68076c2e9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44243 } }
I20260925 09:13:59.370400 7354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:13:59.370570 7354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:13:59.370782 7354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:13:59.373070 7354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82253e063b544eed90b7f2a68076c2e9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44243 } }
I20260925 09:13:59.373615 7354 leader_election.cc:302] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82253e063b544eed90b7f2a68076c2e9; no voters:
I20260925 09:13:59.374182 7354 leader_election.cc:288] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:13:59.374373 7359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:13:59.375134 7359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [term 1 LEADER]: Becoming Leader. State: Replica: 82253e063b544eed90b7f2a68076c2e9, State: Running, Role: LEADER
I20260925 09:13:59.375701 7359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82253e063b544eed90b7f2a68076c2e9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44243 } }
I20260925 09:13:59.376362 7354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:13:59.379038 7361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82253e063b544eed90b7f2a68076c2e9. Latest consensus state: current_term: 1 leader_uuid: "82253e063b544eed90b7f2a68076c2e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82253e063b544eed90b7f2a68076c2e9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44243 } } }
I20260925 09:13:59.379201 7360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82253e063b544eed90b7f2a68076c2e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82253e063b544eed90b7f2a68076c2e9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44243 } } }
I20260925 09:13:59.379488 7361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:59.379593 7360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9 [sys.catalog]: This master's current role is: LEADER
I20260925 09:13:59.381016 7366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:13:59.387377 7366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:13:59.395974 7366 catalog_manager.cc:1347] Generated new cluster ID: bed075faca4341b4a8848995a3bc9290
I20260925 09:13:59.396101 7366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:13:59.408447 7366 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:13:59.410264 7366 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:13:59.423804 7366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9: Generated new TSK 0
I20260925 09:13:59.424700 7366 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:13:59.484843 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:44243
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:13:59.836035 7380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:13:59.836336 7380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:13:59.836506 7380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:13:59.843997 7380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:13:59.844132 7380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:13:59.844317 7380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:13:59.854421 7380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:44243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:13:59.855764 7380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:13:59.857172 7380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:13:59.870731 7386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:59.870767 7388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:13:59.870790 7385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:13:59.871850 7380 server_base.cc:1034] running on GCE node
I20260925 09:13:59.872548 7380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:13:59.873651 7380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:13:59.874858 7380 hybrid_clock.cc:648] HybridClock initialized: now 1790327639874805 us; error 46 us; skew 500 ppm
I20260925 09:13:59.875200 7380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:13:59 dist-test-slave-l6qh krb5kdc[7275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327639, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:13:59.884176 7380 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:13:59.885582 7380 fs_manager.cc:359] Metadata directory not provided
I20260925 09:13:59.885723 7380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:13:59.886088 7380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:13:59.889410 7380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "4b51a05980bf45e2a348dbdca6e5ace0"
format_stamp: "Formatted at 2026-09-25 09:13:59 on dist-test-slave-l6qh"
server_key: "6f3346f97f42a1537c2849646335889a"
server_key_iv: "7891edcb15ca9a0e6a28d54f1ac8debe"
server_key_version: "encryptionkey@0"
I20260925 09:13:59.890285 7380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "4b51a05980bf45e2a348dbdca6e5ace0"
format_stamp: "Formatted at 2026-09-25 09:13:59 on dist-test-slave-l6qh"
server_key: "6f3346f97f42a1537c2849646335889a"
server_key_iv: "7891edcb15ca9a0e6a28d54f1ac8debe"
server_key_version: "encryptionkey@0"
I20260925 09:13:59.895823 7380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:13:59.900998 7394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:59.903294 7380 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260925 09:13:59.903558 7380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "4b51a05980bf45e2a348dbdca6e5ace0"
format_stamp: "Formatted at 2026-09-25 09:13:59 on dist-test-slave-l6qh"
server_key: "6f3346f97f42a1537c2849646335889a"
server_key_iv: "7891edcb15ca9a0e6a28d54f1ac8debe"
server_key_version: "encryptionkey@0"
I20260925 09:13:59.903849 7380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:13:59.939813 7380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:13:59.944686 7380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:13:59.945000 7380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:13:59.946357 7380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:13:59.948339 7380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:13:59.948468 7380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:59.948583 7380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:13:59.948668 7380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:13:59.982266 7380 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:43123
I20260925 09:13:59.982317 7506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:43123 every 8 connection(s)
I20260925 09:13:59.983147 7380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:13:59.987475 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 7380
I20260925 09:13:59.987710 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:13:59.988463 3149 external_mini_cluster.cc:1428] Setting key 45196cd355688b795602634e491fa2b0
Sep 25 09:13:59 dist-test-slave-l6qh krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327639, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:14:00.008001 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:59.985677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:60603 (local address 127.3.19.126:44243)
0925 09:13:59.986139 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 09:13:59.986144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:13:59.987070 (+ 926us) server_negotiation.cc:407] Connection header received
0925 09:13:59.989346 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:13:59.989351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:13:59.989497 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:13:59.989619 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:13:59.991784 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:59.993040 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:59.994805 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:13:59.995530 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:13:59.995690 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:59.999957 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:13:59.999967 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:13:59.999975 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:00.000024 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:00.004153 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:00.005082 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:00.005091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:00.005098 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:00.005318 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:00.005781 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:00.005790 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:00.005796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:00.006256 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:00.006447 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:00.007188 (+ 741us) server_negotiation.cc:299] Negotiation successful
0925 09:14:00.007495 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":262,"threads_started":1}
I20260925 09:14:00.008625 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:13:59.986185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:44243 (local address 127.3.19.65:60603)
0925 09:13:59.986901 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 09:13:59.986940 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 09:13:59.988954 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:13:59.989968 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:13:59.989993 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:13:59.990886 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:13:59.991564 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:59.991586 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:59.993287 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:59.993294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:59.994643 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:13:59.994657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:13:59.995773 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:13:59.995779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:13:59.996073 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:13:59.996787 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:13:59.996811 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:13:59.999691 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:00.004390 (+ 4699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:00.004398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:00.004414 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:00.004904 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:00.005440 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:00.005443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:00.005445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:00.005634 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:00.006405 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:00.006412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:00.006761 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 09:14:00.007005 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 09:14:00.007447 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":303,"threads_started":1}
I20260925 09:14:00.010550 7507 heartbeater.cc:347] Connected to a master server at 127.3.19.126:44243
I20260925 09:14:00.010871 7507 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:00.011559 7507 heartbeater.cc:511] Master 127.3.19.126:44243 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
I20260925 09:14:00.013815 7319 ts_manager.cc:194] Registered new tserver with Master: 4b51a05980bf45e2a348dbdca6e5ace0 (127.3.19.65:43123)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
I20260925 09:14:00.015628 7319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:60603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:14:00.041707 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:44243
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:00.375769 7513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:00.376111 7513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:00.376307 7513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:00.383917 7513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:00.384075 7513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:00.384264 7513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:14:00.394359 7513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:44243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:00.395776 7513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:00.397279 7513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:00.410233 7518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:00.410234 7519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:00.410195 7521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:00.411204 7513 server_base.cc:1034] running on GCE node
I20260925 09:14:00.411829 7513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:00.412737 7513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:00.413951 7513 hybrid_clock.cc:648] HybridClock initialized: now 1790327640413901 us; error 32 us; skew 500 ppm
I20260925 09:14:00.414306 7513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:00 dist-test-slave-l6qh krb5kdc[7275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327640, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:00.417527 7513 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:14:00.418727 7513 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:00.418874 7513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:00.419220 7513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:00.422281 7513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "f2388e66690e45b5b9dfda06e998e349"
format_stamp: "Formatted at 2026-09-25 09:14:00 on dist-test-slave-l6qh"
server_key: "807d493b2fe8c9a493321b213684b3c2"
server_key_iv: "8e6db8802564f9ec069e1aab677e212d"
server_key_version: "encryptionkey@0"
I20260925 09:14:00.423136 7513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "f2388e66690e45b5b9dfda06e998e349"
format_stamp: "Formatted at 2026-09-25 09:14:00 on dist-test-slave-l6qh"
server_key: "807d493b2fe8c9a493321b213684b3c2"
server_key_iv: "8e6db8802564f9ec069e1aab677e212d"
server_key_version: "encryptionkey@0"
I20260925 09:14:00.428433 7513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 09:14:00.432003 7527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:00.433495 7513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:00.433688 7513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "f2388e66690e45b5b9dfda06e998e349"
format_stamp: "Formatted at 2026-09-25 09:14:00 on dist-test-slave-l6qh"
server_key: "807d493b2fe8c9a493321b213684b3c2"
server_key_iv: "8e6db8802564f9ec069e1aab677e212d"
server_key_version: "encryptionkey@0"
I20260925 09:14:00.433916 7513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:00.455650 7513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:00.460004 7513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:00.460300 7513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:00.461356 7513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:00.462723 7513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:00.462846 7513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:00.462987 7513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:00.463078 7513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:00.496672 7513 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:35707
I20260925 09:14:00.496706 7639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:35707 every 8 connection(s)
I20260925 09:14:00.497689 7513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:14:00.502569 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 7513
I20260925 09:14:00.502774 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:14:00.503211 3149 external_mini_cluster.cc:1428] Setting key aa57631105c2e38eb918310b1cae99e8
Sep 25 09:14:00 dist-test-slave-l6qh krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327640, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:14:00.519829 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:00.500265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:51871 (local address 127.3.19.126:44243)
0925 09:14:00.500462 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:00.500467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:00.501741 (+ 1274us) server_negotiation.cc:407] Connection header received
0925 09:14:00.503765 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:00.503770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:00.503872 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:00.503999 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:00.505627 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:00.506327 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:00.507910 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:00.508368 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:00.508521 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:00.512459 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:00.512469 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:00.512478 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:00.512526 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:00.516275 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:00.517131 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:00.517139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:00.517145 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:00.517337 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:00.517786 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:00.517793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:00.517800 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:00.518188 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:00.518345 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:00.519086 (+ 741us) server_negotiation.cc:299] Negotiation successful
0925 09:14:00.519356 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 09:14:00.520642 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:00.500719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:44243 (local address 127.3.19.66:51871)
0925 09:14:00.501566 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:00.501610 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:00.503475 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:00.504238 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:00.504247 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:00.504789 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:00.505455 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:00.505476 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:00.506489 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:00.506493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:00.507742 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:00.507754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:00.508617 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:00.508624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:00.508862 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:00.509654 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:00.509701 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:00.512183 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:00.516470 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:00.516478 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:00.516493 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:00.516976 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:00.517472 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:00.517476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:00.517478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:00.517665 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:00.518318 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:00.518324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:00.518683 (+ 359us) client_negotiation.cc:769] Sending connection context
0925 09:14:00.518925 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 09:14:00.519367 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":332,"threads_started":1}
I20260925 09:14:00.522346 7640 heartbeater.cc:347] Connected to a master server at 127.3.19.126:44243
I20260925 09:14:00.522691 7640 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:00.523353 7640 heartbeater.cc:511] Master 127.3.19.126:44243 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
I20260925 09:14:00.525022 7319 ts_manager.cc:194] Registered new tserver with Master: f2388e66690e45b5b9dfda06e998e349 (127.3.19.66:35707)
I20260925 09:14:00.525949 7319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:51871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:14:00.553664 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:44243
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:00.886890 7645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:00.887202 7645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:00.887300 7645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:00.894995 7645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:00.895123 7645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:00.895242 7645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:14:00.905344 7645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:44243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:00.906651 7645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:00.907905 7645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:00.921478 7653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:00.921514 7650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:00.921478 7651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:00.922286 7645 server_base.cc:1034] running on GCE node
I20260925 09:14:00.922842 7645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:00.923775 7645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:00.925037 7645 hybrid_clock.cc:648] HybridClock initialized: now 1790327640924994 us; error 52 us; skew 500 ppm
I20260925 09:14:00.925518 7645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:00 dist-test-slave-l6qh krb5kdc[7275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327640, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:00.928665 7645 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:14:00.929960 7645 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:00.930099 7645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:00.930429 7645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:00.933476 7645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "c01b48663da844639cbb51d74049b074"
format_stamp: "Formatted at 2026-09-25 09:14:00 on dist-test-slave-l6qh"
server_key: "637a297772aa860e25cf648dbbc4b003"
server_key_iv: "b324492863b60f2745b82ae9a7c61d65"
server_key_version: "encryptionkey@0"
I20260925 09:14:00.934314 7645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "c01b48663da844639cbb51d74049b074"
format_stamp: "Formatted at 2026-09-25 09:14:00 on dist-test-slave-l6qh"
server_key: "637a297772aa860e25cf648dbbc4b003"
server_key_iv: "b324492863b60f2745b82ae9a7c61d65"
server_key_version: "encryptionkey@0"
I20260925 09:14:00.939615 7645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 09:14:00.943225 7659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:00.944587 7645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:14:00.944773 7645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "c01b48663da844639cbb51d74049b074"
format_stamp: "Formatted at 2026-09-25 09:14:00 on dist-test-slave-l6qh"
server_key: "637a297772aa860e25cf648dbbc4b003"
server_key_iv: "b324492863b60f2745b82ae9a7c61d65"
server_key_version: "encryptionkey@0"
I20260925 09:14:00.945005 7645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:00.966549 7645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:00.970737 7645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:00.971042 7645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:00.972060 7645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:00.973534 7645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:00.973668 7645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:00.973795 7645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:00.973888 7645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:01.008972 7645 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:33833
I20260925 09:14:01.009064 7771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:33833 every 8 connection(s)
I20260925 09:14:01.009871 7645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:14:01.014822 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 7645
I20260925 09:14:01.015053 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:14:01.015573 3149 external_mini_cluster.cc:1428] Setting key 4950035d5880ac240fe54ea791ee9a29
I20260925 09:14:01.018970 7507 heartbeater.cc:503] Master 127.3.19.126:44243 was elected leader, sending a full tablet report...
Sep 25 09:14:01 dist-test-slave-l6qh krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327640, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:14:01.031404 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.012538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:57993 (local address 127.3.19.126:44243)
0925 09:14:01.012736 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:01.012742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:01.014024 (+ 1282us) server_negotiation.cc:407] Connection header received
0925 09:14:01.016446 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:01.016452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:01.016566 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:01.016684 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:01.018715 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.019633 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.021585 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.021979 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.022075 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.025521 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:01.025525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:01.025528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:01.025557 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:01.028002 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:01.028730 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:01.028734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:01.028737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:01.028862 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:01.029388 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:01.029392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:01.029394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:01.029729 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:01.029890 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:01.030784 (+ 894us) server_negotiation.cc:299] Negotiation successful
0925 09:14:01.031001 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 09:14:01.032447 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.013047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:44243 (local address 127.3.19.67:57993)
0925 09:14:01.013856 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:01.013901 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:01.016180 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:01.016958 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:01.016967 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:01.017731 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:01.018551 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:01.018570 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.019784 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.019790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:01.021442 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:01.021459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.022182 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.022187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:01.022370 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.023099 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:01.023121 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:01.025290 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:01.028134 (+ 2844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:01.028141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:01.028156 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:01.028604 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:01.028992 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:01.028998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:01.029000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:01.029233 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:01.029870 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:01.029878 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:01.030345 (+ 467us) client_negotiation.cc:769] Sending connection context
0925 09:14:01.030699 (+ 354us) client_negotiation.cc:240] Negotiation successful
0925 09:14:01.031295 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":406,"threads_started":1}
I20260925 09:14:01.034040 7772 heartbeater.cc:347] Connected to a master server at 127.3.19.126:44243
I20260925 09:14:01.034399 7772 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:01.035115 7772 heartbeater.cc:511] Master 127.3.19.126:44243 requested a full tablet report, sending...
I20260925 09:14:01.036664 7319 ts_manager.cc:194] Registered new tserver with Master: c01b48663da844639cbb51d74049b074 (127.3.19.67:33833)
I20260925 09:14:01.037711 7319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:57993
I20260925 09:14:01.043715 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:14:01.060433 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.048216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.3.19.126:44243)
0925 09:14:01.048420 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:01.048429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:01.048969 (+ 540us) server_negotiation.cc:407] Connection header received
0925 09:14:01.049111 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:01.049115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:01.049178 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:01.049346 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:01.050787 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.051519 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.052673 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.053088 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.053184 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.055453 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:01.055457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:01.055460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:01.055488 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:01.057847 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:01.058419 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:01.058422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:01.058425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:01.058545 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:01.058937 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:01.058944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:01.058947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:01.059287 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:01.059546 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:01.059880 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 09:14:01.060108 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 09:14:01.064703 7319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:14:01.071501 7319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:14:01.096501 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.083430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:35707 (local address 127.0.0.1:37476)
0925 09:14:01.084272 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:01.084321 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:01.084599 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:01.085898 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:01.085909 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:01.085948 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:01.086428 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:01.086446 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.089217 (+ 2771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.089257 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:01.092596 (+ 3339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:01.092622 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.095354 (+ 2732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.095364 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:01.095630 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.095690 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 09:14:01.095812 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 09:14:01.096026 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":344,"threads_started":1}
I20260925 09:14:01.097751 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.083700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.3.19.66:35707)
0925 09:14:01.085209 (+ 1509us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:01.085217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:01.085273 (+ 56us) server_negotiation.cc:407] Connection header received
0925 09:14:01.085374 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:01.085383 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:01.085652 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:01.086678 (+ 1026us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:01.086959 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.089012 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.092913 (+ 3901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.095049 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.096868 (+ 1819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.097118 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:01.097199 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 09:14:01.097406 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1362,"thread_start_us":1252,"threads_started":1}
I20260925 09:14:01.102037 7575 tablet_service.cc:1467] Processing CreateTablet for tablet fee30172df1b47f296781d31ab179269 (DEFAULT_TABLE table=test-table [id=c3e3ab00d65349b9850e6eca1b5e0d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:01.103447 7575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee30172df1b47f296781d31ab179269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:01.106971 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.086270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.3.19.67:33833)
0925 09:14:01.098712 (+ 12442us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:01.098718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:01.098781 (+ 63us) server_negotiation.cc:407] Connection header received
0925 09:14:01.098889 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:01.098897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:01.099171 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:01.099685 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:01.100236 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.101697 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.104272 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.105840 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.105992 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.106222 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:01.106453 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 09:14:01.106621 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12283,"thread_start_us":12133,"threads_started":1}
I20260925 09:14:01.107005 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.086003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:33833 (local address 127.0.0.1:47752)
0925 09:14:01.096654 (+ 10651us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:01.096672 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:01.096971 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:01.099392 (+ 2421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:01.099397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:01.099428 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:01.099994 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:01.100007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.101837 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.101842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:01.104054 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:01.104070 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.106054 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.106060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:01.106230 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.106277 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 09:14:01.106388 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 09:14:01.106547 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10486,"mutex_wait_us":52,"thread_start_us":9963,"threads_started":1}
I20260925 09:14:01.107507 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.083664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:43123 (local address 127.0.0.1:34398)
0925 09:14:01.095736 (+ 12072us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:01.095758 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:01.095878 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:01.099280 (+ 3402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:01.099284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:01.099310 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:01.099871 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:01.099891 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.102162 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.102167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:01.104268 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:01.104278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.106406 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.106413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:01.106581 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.106613 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 09:14:01.106714 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 09:14:01.106854 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11927,"mutex_wait_us":57,"thread_start_us":12675,"threads_started":1}
I20260925 09:14:01.108182 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.084311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.3.19.65:43123)
0925 09:14:01.097766 (+ 13455us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:01.097777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:01.097811 (+ 34us) server_negotiation.cc:407] Connection header received
0925 09:14:01.098847 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:01.098854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:01.099084 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:01.099294 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:01.100093 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.102020 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.104438 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.105848 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.105992 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.106227 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:01.107822 (+ 1595us) server_negotiation.cc:299] Negotiation successful
0925 09:14:01.107963 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13279,"thread_start_us":12958,"threads_started":1}
I20260925 09:14:01.111533 7707 tablet_service.cc:1467] Processing CreateTablet for tablet fee30172df1b47f296781d31ab179269 (DEFAULT_TABLE table=test-table [id=c3e3ab00d65349b9850e6eca1b5e0d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:01.111629 7442 tablet_service.cc:1467] Processing CreateTablet for tablet fee30172df1b47f296781d31ab179269 (DEFAULT_TABLE table=test-table [id=c3e3ab00d65349b9850e6eca1b5e0d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:01.113026 7442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee30172df1b47f296781d31ab179269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:01.113027 7707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee30172df1b47f296781d31ab179269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:01.117038 7790 tablet_bootstrap.cc:492] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349: Bootstrap starting.
I20260925 09:14:01.120635 7790 tablet_bootstrap.cc:654] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:01.121934 7790 log.cc:824] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:01.124647 7790 tablet_bootstrap.cc:492] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349: No bootstrap required, opened a new log
I20260925 09:14:01.124959 7790 ts_tablet_manager.cc:1397] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 09:14:01.129176 7792 tablet_bootstrap.cc:492] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074: Bootstrap starting.
I20260925 09:14:01.130049 7793 tablet_bootstrap.cc:492] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0: Bootstrap starting.
I20260925 09:14:01.133093 7792 tablet_bootstrap.cc:654] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:01.133538 7793 tablet_bootstrap.cc:654] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:01.134259 7792 log.cc:824] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:01.134784 7793 log.cc:824] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:01.136297 7790 raft_consensus.cc:348] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2388e66690e45b5b9dfda06e998e349" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 35707 } } peers { permanent_uuid: "4b51a05980bf45e2a348dbdca6e5ace0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 43123 } } peers { permanent_uuid: "c01b48663da844639cbb51d74049b074" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33833 } }
I20260925 09:14:01.136904 7790 raft_consensus.cc:374] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:01.137059 7790 raft_consensus.cc:729] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2388e66690e45b5b9dfda06e998e349, State: Initialized, Role: FOLLOWER
I20260925 09:14:01.137730 7792 tablet_bootstrap.cc:492] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074: No bootstrap required, opened a new log
I20260925 09:14:01.137749 7793 tablet_bootstrap.cc:492] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0: No bootstrap required, opened a new log
I20260925 09:14:01.138020 7792 ts_tablet_manager.cc:1397] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 09:14:01.138031 7793 ts_tablet_manager.cc:1397] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 09:14:01.137933 7790 consensus_queue.cc:260] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2388e66690e45b5b9dfda06e998e349" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 35707 } } peers { permanent_uuid: "4b51a05980bf45e2a348dbdca6e5ace0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 43123 } } peers { permanent_uuid: "c01b48663da844639cbb51d74049b074" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33833 } }
I20260925 09:14:01.139153 7640 heartbeater.cc:503] Master 127.3.19.126:44243 was elected leader, sending a full tablet report...
I20260925 09:14:01.139887 7790 ts_tablet_manager.cc:1428] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349: Time spent starting tablet: real 0.015s user 0.004s sys 0.010s
I20260925 09:14:01.147372 7793 raft_consensus.cc:348] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2388e66690e45b5b9dfda06e998e349" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 35707 } } peers { permanent_uuid: "4b51a05980bf45e2a348dbdca6e5ace0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 43123 } } peers { permanent_uuid: "c01b48663da844639cbb51d74049b074" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33833 } }
I20260925 09:14:01.147864 7793 raft_consensus.cc:374] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:01.147979 7793 raft_consensus.cc:729] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b51a05980bf45e2a348dbdca6e5ace0, State: Initialized, Role: FOLLOWER
I20260925 09:14:01.147728 7792 raft_consensus.cc:348] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2388e66690e45b5b9dfda06e998e349" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 35707 } } peers { permanent_uuid: "4b51a05980bf45e2a348dbdca6e5ace0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 43123 } } peers { permanent_uuid: "c01b48663da844639cbb51d74049b074" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33833 } }
I20260925 09:14:01.148250 7792 raft_consensus.cc:374] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:01.148341 7792 raft_consensus.cc:729] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c01b48663da844639cbb51d74049b074, State: Initialized, Role: FOLLOWER
I20260925 09:14:01.148653 7793 consensus_queue.cc:260] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2388e66690e45b5b9dfda06e998e349" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 35707 } } peers { permanent_uuid: "4b51a05980bf45e2a348dbdca6e5ace0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 43123 } } peers { permanent_uuid: "c01b48663da844639cbb51d74049b074" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33833 } }
I20260925 09:14:01.148808 7792 consensus_queue.cc:260] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2388e66690e45b5b9dfda06e998e349" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 35707 } } peers { permanent_uuid: "4b51a05980bf45e2a348dbdca6e5ace0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 43123 } } peers { permanent_uuid: "c01b48663da844639cbb51d74049b074" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33833 } }
I20260925 09:14:01.149863 7772 heartbeater.cc:503] Master 127.3.19.126:44243 was elected leader, sending a full tablet report...
I20260925 09:14:01.150553 7792 ts_tablet_manager.cc:1428] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260925 09:14:01.150557 7793 ts_tablet_manager.cc:1428] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
W20260925 09:14:01.235667 7508 tablet.cc:2369] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:14:01.249871 7641 tablet.cc:2369] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:14:01.261482 7773 tablet.cc:2369] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:14:01.363816 7796 raft_consensus.cc:482] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:14:01.364197 7796 raft_consensus.cc:504] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2388e66690e45b5b9dfda06e998e349" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 35707 } } peers { permanent_uuid: "4b51a05980bf45e2a348dbdca6e5ace0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 43123 } } peers { permanent_uuid: "c01b48663da844639cbb51d74049b074" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33833 } }
I20260925 09:14:01.365996 7796 leader_election.cc:288] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b51a05980bf45e2a348dbdca6e5ace0 (127.3.19.65:43123), c01b48663da844639cbb51d74049b074 (127.3.19.67:33833)
I20260925 09:14:01.372706 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.366544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:34255 (local address 127.3.19.67:33833)
0925 09:14:01.366723 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:01.366728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:01.367424 (+ 696us) server_negotiation.cc:407] Connection header received
0925 09:14:01.367662 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:01.367668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:01.367738 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:01.367839 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:01.368411 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.369145 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.370922 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.371743 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.371902 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.372012 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:01.372297 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 09:14:01.372424 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 09:14:01.373124 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.366585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:33833 (local address 127.3.19.66:34255)
0925 09:14:01.367291 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:01.367308 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:01.367507 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:01.367863 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:01.367867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:01.367894 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:01.368264 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:01.368273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.369294 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.369299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:01.370760 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:01.370772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.371967 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.371973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:01.372136 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.372159 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 09:14:01.372270 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 09:14:01.372414 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":1,"spinlock_wait_cycles":896,"thread_start_us":931,"threads_started":1}
I20260925 09:14:01.373752 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.366479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:45993 (local address 127.3.19.65:43123)
0925 09:14:01.366714 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:01.366722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:01.368012 (+ 1290us) server_negotiation.cc:407] Connection header received
0925 09:14:01.368276 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:01.368288 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:01.368354 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:01.368474 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:01.369055 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.369913 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.371790 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.372657 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.372826 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.373023 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:01.373464 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 09:14:01.373585 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 09:14:01.374027 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.366631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:43123 (local address 127.3.19.66:45993)
0925 09:14:01.367868 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:01.367891 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:01.368104 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:01.368490 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:01.368495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:01.368520 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:01.368877 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:01.368893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.370068 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.370074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:01.371618 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:01.371630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.373023 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.373030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:01.373238 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.373311 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 09:14:01.373418 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 09:14:01.373578 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"spinlock_wait_cycles":1280,"thread_start_us":393,"threads_started":1}
I20260925 09:14:01.374429 7727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee30172df1b47f296781d31ab179269" candidate_uuid: "f2388e66690e45b5b9dfda06e998e349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c01b48663da844639cbb51d74049b074" is_pre_election: true
I20260925 09:14:01.375109 7727 raft_consensus.cc:2454] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2388e66690e45b5b9dfda06e998e349 in term 0.
I20260925 09:14:01.375104 7462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee30172df1b47f296781d31ab179269" candidate_uuid: "f2388e66690e45b5b9dfda06e998e349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b51a05980bf45e2a348dbdca6e5ace0" is_pre_election: true
I20260925 09:14:01.375662 7462 raft_consensus.cc:2454] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2388e66690e45b5b9dfda06e998e349 in term 0.
I20260925 09:14:01.375876 7528 leader_election.cc:302] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c01b48663da844639cbb51d74049b074, f2388e66690e45b5b9dfda06e998e349; no voters:
I20260925 09:14:01.376318 7796 raft_consensus.cc:2781] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:14:01.376500 7796 raft_consensus.cc:482] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:14:01.376632 7796 raft_consensus.cc:3037] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:01.379163 7796 raft_consensus.cc:504] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2388e66690e45b5b9dfda06e998e349" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 35707 } } peers { permanent_uuid: "4b51a05980bf45e2a348dbdca6e5ace0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 43123 } } peers { permanent_uuid: "c01b48663da844639cbb51d74049b074" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33833 } }
I20260925 09:14:01.379799 7796 leader_election.cc:288] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [CANDIDATE]: Term 1 election: Requested vote from peers 4b51a05980bf45e2a348dbdca6e5ace0 (127.3.19.65:43123), c01b48663da844639cbb51d74049b074 (127.3.19.67:33833)
I20260925 09:14:01.380366 7727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee30172df1b47f296781d31ab179269" candidate_uuid: "f2388e66690e45b5b9dfda06e998e349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c01b48663da844639cbb51d74049b074"
I20260925 09:14:01.380366 7462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee30172df1b47f296781d31ab179269" candidate_uuid: "f2388e66690e45b5b9dfda06e998e349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b51a05980bf45e2a348dbdca6e5ace0"
I20260925 09:14:01.380604 7727 raft_consensus.cc:3037] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:01.380625 7462 raft_consensus.cc:3037] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:01.384315 7462 raft_consensus.cc:2454] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2388e66690e45b5b9dfda06e998e349 in term 1.
I20260925 09:14:01.384317 7727 raft_consensus.cc:2454] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2388e66690e45b5b9dfda06e998e349 in term 1.
I20260925 09:14:01.384862 7529 leader_election.cc:302] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b51a05980bf45e2a348dbdca6e5ace0, f2388e66690e45b5b9dfda06e998e349; no voters:
I20260925 09:14:01.385263 7796 raft_consensus.cc:2781] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:01.385735 7796 raft_consensus.cc:686] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [term 1 LEADER]: Becoming Leader. State: Replica: f2388e66690e45b5b9dfda06e998e349, State: Running, Role: LEADER
I20260925 09:14:01.386262 7796 consensus_queue.cc:237] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2388e66690e45b5b9dfda06e998e349" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 35707 } } peers { permanent_uuid: "4b51a05980bf45e2a348dbdca6e5ace0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 43123 } } peers { permanent_uuid: "c01b48663da844639cbb51d74049b074" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33833 } }
I20260925 09:14:01.391505 7319 catalog_manager.cc:5495] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2388e66690e45b5b9dfda06e998e349 (127.3.19.66). New cstate: current_term: 1 leader_uuid: "f2388e66690e45b5b9dfda06e998e349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2388e66690e45b5b9dfda06e998e349" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 35707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b51a05980bf45e2a348dbdca6e5ace0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 43123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c01b48663da844639cbb51d74049b074" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33833 } health_report { overall_health: UNKNOWN } } }
I20260925 09:14:01.476727 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.471167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37482 (local address 127.3.19.66:35707)
0925 09:14:01.471475 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:01.471482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:01.471529 (+ 47us) server_negotiation.cc:407] Connection header received
0925 09:14:01.471803 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:01.471808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:01.471886 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:01.472046 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:14:01.472409 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.473125 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.474553 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.474960 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.475090 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.475579 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:14:01.476133 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:14:01.476324 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:01.476380 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 09:14:01.476465 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 09:14:01.485749 7462 raft_consensus.cc:1261] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0 [term 1 FOLLOWER]: Refusing update from remote peer f2388e66690e45b5b9dfda06e998e349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:01.485749 7727 raft_consensus.cc:1261] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074 [term 1 FOLLOWER]: Refusing update from remote peer f2388e66690e45b5b9dfda06e998e349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:01.486986 7796 consensus_queue.cc:1035] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c01b48663da844639cbb51d74049b074" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:01.487401 7801 consensus_queue.cc:1035] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b51a05980bf45e2a348dbdca6e5ace0" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 43123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:01.502606 7804 mvcc.cc:205] Tried to move back new op lower bound from 7333182019513712640 to 7333182019121971200. Current Snapshot: MvccSnapshot[applied={T|T < 7333182019513712640}]
I20260925 09:14:01.526698 7319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:01.526983 7319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:01.530668 7319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82253e063b544eed90b7f2a68076c2e9: Sending DeleteTablet for 3 replicas of tablet fee30172df1b47f296781d31ab179269
I20260925 09:14:01.531919 7707 tablet_service.cc:1514] Processing DeleteTablet for tablet fee30172df1b47f296781d31ab179269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:01 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:47752
I20260925 09:14:01.531944 7442 tablet_service.cc:1514] Processing DeleteTablet for tablet fee30172df1b47f296781d31ab179269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:01 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:34398
I20260925 09:14:01.532111 7575 tablet_service.cc:1514] Processing DeleteTablet for tablet fee30172df1b47f296781d31ab179269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:01 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:37476
I20260925 09:14:01.534204 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 7380
I20260925 09:14:01.544615 7813 tablet_replica.cc:330] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074: stopping tablet replica
I20260925 09:14:01.545321 7813 raft_consensus.cc:2229] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:01.546015 7813 raft_consensus.cc:2258] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:01.546995 7814 tablet_replica.cc:330] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0: stopping tablet replica
I20260925 09:14:01.548312 7815 tablet_replica.cc:330] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349: stopping tablet replica
I20260925 09:14:01.548770 7815 raft_consensus.cc:2229] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:14:01.549558 7815 raft_consensus.cc:2258] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:01.547573 7814 raft_consensus.cc:2229] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:01.550238 7813 ts_tablet_manager.cc:1905] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:01.550318 7814 raft_consensus.cc:2258] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:01.552698 7814 ts_tablet_manager.cc:1905] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:01.555045 7815 ts_tablet_manager.cc:1905] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:01.557750 7813 ts_tablet_manager.cc:1918] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:01.557929 7813 log.cc:1196] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/fee30172df1b47f296781d31ab179269
I20260925 09:14:01.558419 7813 ts_tablet_manager.cc:1939] T fee30172df1b47f296781d31ab179269 P c01b48663da844639cbb51d74049b074: Deleting consensus metadata
I20260925 09:14:01.560396 7303 catalog_manager.cc:4841] TS c01b48663da844639cbb51d74049b074 (127.3.19.67:33833): tablet fee30172df1b47f296781d31ab179269 (table test-table [id=c3e3ab00d65349b9850e6eca1b5e0d50]) successfully deleted
I20260925 09:14:01.562532 7815 ts_tablet_manager.cc:1918] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:01.562713 7815 log.cc:1196] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/fee30172df1b47f296781d31ab179269
I20260925 09:14:01.563326 7815 ts_tablet_manager.cc:1939] T fee30172df1b47f296781d31ab179269 P f2388e66690e45b5b9dfda06e998e349: Deleting consensus metadata
I20260925 09:14:01.565328 7306 catalog_manager.cc:4841] TS f2388e66690e45b5b9dfda06e998e349 (127.3.19.66:35707): tablet fee30172df1b47f296781d31ab179269 (table test-table [id=c3e3ab00d65349b9850e6eca1b5e0d50]) successfully deleted
Sep 25 09:14:01 dist-test-slave-l6qh krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:14:01.566583 7814 ts_tablet_manager.cc:1918] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:01.566782 7814 log.cc:1196] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/fee30172df1b47f296781d31ab179269
I20260925 09:14:01.567384 7814 ts_tablet_manager.cc:1939] T fee30172df1b47f296781d31ab179269 P 4b51a05980bf45e2a348dbdca6e5ace0: Deleting consensus metadata
I20260925 09:14:01.569530 7304 catalog_manager.cc:4841] TS 4b51a05980bf45e2a348dbdca6e5ace0 (127.3.19.65:43123): tablet fee30172df1b47f296781d31ab179269 (table test-table [id=c3e3ab00d65349b9850e6eca1b5e0d50]) successfully deleted
I20260925 09:14:01.573478 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.535772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.3.19.65:43123)
0925 09:14:01.535955 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:01.535961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:01.550293 (+ 14332us) server_negotiation.cc:407] Connection header received
0925 09:14:01.550986 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:01.550992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:01.551074 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:01.551172 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:01.555301 (+ 4129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.556430 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.561335 (+ 4905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.562012 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.562124 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.567311 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:01.567324 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:01.567339 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:01.567383 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:01.570613 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:01.571371 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:01.571378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:01.571380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:01.571520 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:01.571916 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:01.571921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:01.571923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:01.572270 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:01.572605 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:01.572880 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 09:14:01.573089 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 09:14:01.574139 7502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:01.710215 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 7380
I20260925 09:14:01.728108 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 7513
Sep 25 09:14:01 dist-test-slave-l6qh krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:14:01.744590 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.728908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.3.19.66:35707)
0925 09:14:01.729073 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:01.729081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:01.729101 (+ 20us) server_negotiation.cc:407] Connection header received
0925 09:14:01.729345 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:01.729351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:01.729431 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:01.729539 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:01.730724 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.731566 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.732760 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.733221 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.733544 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.737971 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:01.737986 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:01.738047 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:01.738108 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:01.741461 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:01.742248 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:01.742263 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:01.742266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:01.742421 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:01.742855 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:01.742862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:01.742867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:01.743276 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:01.743659 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:01.743943 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 09:14:01.744178 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 09:14:01.745551 7635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:01.861845 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 7513
I20260925 09:14:01.880954 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 7645
Sep 25 09:14:01 dist-test-slave-l6qh krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:14:01.897927 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:01.881904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47754 (local address 127.3.19.67:33833)
0925 09:14:01.882434 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:01.882439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:01.882455 (+ 16us) server_negotiation.cc:407] Connection header received
0925 09:14:01.882513 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:01.882516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:01.882585 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:01.882730 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:01.884169 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.884908 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.886490 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:01.886934 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:01.887085 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:01.891888 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:01.891909 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:01.891930 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:01.891980 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:01.895065 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:01.895816 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:01.895824 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:01.895826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:01.895978 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:01.896328 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:01.896331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:01.896332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:01.896674 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:01.896960 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:01.897421 (+ 461us) server_negotiation.cc:299] Negotiation successful
0925 09:14:01.897658 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":236,"threads_started":1}
I20260925 09:14:01.898602 7767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:01.987136 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 7645
I20260925 09:14:02.003782 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 7288
I20260925 09:14:02.004982 7349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:02.112406 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 7288
2026-09-25T09:14:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4523 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T09:14:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:14:02Z Disabled control of system clock
I20260925 09:14:02.181396 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:45869
--webserver_interface=127.3.19.126
--webserver_port=0
--builtin_ntp_servers=127.3.19.84:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:45869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 09:14:02.513547 7825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:02.513948 7825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:02.514103 7825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:02.521584 7825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:02.521755 7825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:02.521848 7825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:02.521967 7825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:02.522045 7825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:14:02.531570 7825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:45869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:45869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:02.532972 7825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:02.534822 7825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:02.547133 7833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:02.547133 7830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:02.547166 7831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:02.548198 7825 server_base.cc:1034] running on GCE node
I20260925 09:14:02.549084 7825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:02.550645 7825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:02.551877 7825 hybrid_clock.cc:648] HybridClock initialized: now 1790327642551832 us; error 65 us; skew 500 ppm
I20260925 09:14:02.552258 7825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:02.554729 7825 webserver.cc:460] Webserver started at http://127.3.19.126:36749/ using document root <none> and password file <none>
I20260925 09:14:02.555819 7825 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:02.555956 7825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:02.556309 7825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:02.560014 7825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "8fe86b5cd5f84f1baeb4898b7d2fe5c9"
format_stamp: "Formatted at 2026-09-25 09:14:02 on dist-test-slave-l6qh"
server_key: "0feb2a17d7c4d2089989ab564b4b054b"
server_key_iv: "50b469f8d7066ea2872651d28424ccbc"
server_key_version: "encryptionkey@0"
I20260925 09:14:02.560847 7825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "8fe86b5cd5f84f1baeb4898b7d2fe5c9"
format_stamp: "Formatted at 2026-09-25 09:14:02 on dist-test-slave-l6qh"
server_key: "0feb2a17d7c4d2089989ab564b4b054b"
server_key_iv: "50b469f8d7066ea2872651d28424ccbc"
server_key_version: "encryptionkey@0"
I20260925 09:14:02.566325 7825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 09:14:02.569971 7839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:02.571653 7825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:14:02.571817 7825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "8fe86b5cd5f84f1baeb4898b7d2fe5c9"
format_stamp: "Formatted at 2026-09-25 09:14:02 on dist-test-slave-l6qh"
server_key: "0feb2a17d7c4d2089989ab564b4b054b"
server_key_iv: "50b469f8d7066ea2872651d28424ccbc"
server_key_version: "encryptionkey@0"
I20260925 09:14:02.572023 7825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:02.578114 7825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:02.578922 7825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:02.579207 7825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:02.598354 7825 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:45869
I20260925 09:14:02.598374 7890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:45869 every 8 connection(s)
I20260925 09:14:02.599771 7825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:14:02.601980 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 7825
I20260925 09:14:02.602182 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:14:02.602888 3149 external_mini_cluster.cc:1428] Setting key 25c1003dfdeef822b3a3817c61612f61
I20260925 09:14:02.603965 7891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:02.612970 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:02.605060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.3.19.126:45869)
0925 09:14:02.606224 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:02.606240 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:02.606292 (+ 52us) server_negotiation.cc:407] Connection header received
0925 09:14:02.608705 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:02.608738 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:02.609366 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:02.609863 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:02.610933 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:02.610943 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:02.610961 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:02.610977 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:02.611052 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:02.611074 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:02.611314 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:02.611468 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 09:14:02.611755 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":426,"threads_started":1}
I20260925 09:14:02.618247 7891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9: Bootstrap starting.
I20260925 09:14:02.621112 7891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:02.622291 7891 log.cc:824] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:02.625005 7891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9: No bootstrap required, opened a new log
I20260925 09:14:02.632537 7891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe86b5cd5f84f1baeb4898b7d2fe5c9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 45869 } }
I20260925 09:14:02.632999 7891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:02.633127 7891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fe86b5cd5f84f1baeb4898b7d2fe5c9, State: Initialized, Role: FOLLOWER
I20260925 09:14:02.633752 7891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe86b5cd5f84f1baeb4898b7d2fe5c9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 45869 } }
I20260925 09:14:02.633992 7891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:14:02.634122 7891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:14:02.634287 7891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:02.636965 7891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe86b5cd5f84f1baeb4898b7d2fe5c9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 45869 } }
I20260925 09:14:02.637542 7891 leader_election.cc:302] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fe86b5cd5f84f1baeb4898b7d2fe5c9; no voters:
I20260925 09:14:02.638199 7891 leader_election.cc:288] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:14:02.638338 7896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:02.639107 7896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [term 1 LEADER]: Becoming Leader. State: Replica: 8fe86b5cd5f84f1baeb4898b7d2fe5c9, State: Running, Role: LEADER
I20260925 09:14:02.639691 7896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe86b5cd5f84f1baeb4898b7d2fe5c9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 45869 } }
I20260925 09:14:02.640393 7891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:02.642737 7898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fe86b5cd5f84f1baeb4898b7d2fe5c9. Latest consensus state: current_term: 1 leader_uuid: "8fe86b5cd5f84f1baeb4898b7d2fe5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe86b5cd5f84f1baeb4898b7d2fe5c9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 45869 } } }
I20260925 09:14:02.643229 7898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:02.643834 7897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fe86b5cd5f84f1baeb4898b7d2fe5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe86b5cd5f84f1baeb4898b7d2fe5c9" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 45869 } } }
I20260925 09:14:02.644188 7897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:02.644842 7902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:14:02.649454 7902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:14:02.658429 7902 catalog_manager.cc:1347] Generated new cluster ID: 32d3eed1161c4ebea707aab48caacb07
I20260925 09:14:02.658555 7902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:14:02.703965 7902 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:14:02.704504 7902 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:14:02.714555 7902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9: Generated new TSK 0
I20260925 09:14:02.715571 7902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:14:02.730425 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:45869
--builtin_ntp_servers=127.3.19.84:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 09:14:03.066936 7915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:03.067245 7915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:03.067420 7915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:03.075009 7915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:03.075155 7915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:03.075351 7915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:14:03.085139 7915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:03.086637 7915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:03.088510 7915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:03.102362 7920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:03.102406 7921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:03.102455 7923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:03.103816 7915 server_base.cc:1034] running on GCE node
I20260925 09:14:03.104575 7915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:03.105752 7915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:03.106999 7915 hybrid_clock.cc:648] HybridClock initialized: now 1790327643106928 us; error 69 us; skew 500 ppm
I20260925 09:14:03.107358 7915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:03.109751 7915 webserver.cc:460] Webserver started at http://127.3.19.65:42339/ using document root <none> and password file <none>
I20260925 09:14:03.110491 7915 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:03.110608 7915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:03.110952 7915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:03.113951 7915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "aeb6af4b5f1e4587a0db1f19116bcdec"
format_stamp: "Formatted at 2026-09-25 09:14:03 on dist-test-slave-l6qh"
server_key: "9a24f0435ff451368c2d53ebabc7ad63"
server_key_iv: "fe52b360450e7161e5365b55f6e0dfd3"
server_key_version: "encryptionkey@0"
I20260925 09:14:03.114766 7915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "aeb6af4b5f1e4587a0db1f19116bcdec"
format_stamp: "Formatted at 2026-09-25 09:14:03 on dist-test-slave-l6qh"
server_key: "9a24f0435ff451368c2d53ebabc7ad63"
server_key_iv: "fe52b360450e7161e5365b55f6e0dfd3"
server_key_version: "encryptionkey@0"
I20260925 09:14:03.120095 7915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:14:03.123872 7929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:03.125375 7915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 09:14:03.125566 7915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "aeb6af4b5f1e4587a0db1f19116bcdec"
format_stamp: "Formatted at 2026-09-25 09:14:03 on dist-test-slave-l6qh"
server_key: "9a24f0435ff451368c2d53ebabc7ad63"
server_key_iv: "fe52b360450e7161e5365b55f6e0dfd3"
server_key_version: "encryptionkey@0"
I20260925 09:14:03.125823 7915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:03.131690 7915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:03.132596 7915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:03.132889 7915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:03.133894 7915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:03.135280 7915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:03.135406 7915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:03.135540 7915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:03.135628 7915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:03.169201 7915 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:42477
I20260925 09:14:03.169310 8041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:42477 every 8 connection(s)
I20260925 09:14:03.170549 7915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:14:03.171257 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 7915
I20260925 09:14:03.171480 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:14:03.172256 3149 external_mini_cluster.cc:1428] Setting key b00eda6975de7b1ca60779c181ed8749
I20260925 09:14:03.175379 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:45869
--builtin_ntp_servers=127.3.19.84:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 09:14:03.179601 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:03.173392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:48567 (local address 127.3.19.126:45869)
0925 09:14:03.173844 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:03.173848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:03.175181 (+ 1333us) server_negotiation.cc:407] Connection header received
0925 09:14:03.177213 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:03.177218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:03.177320 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:03.177451 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:03.178818 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:03.178823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:03.178828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:03.178847 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:03.178902 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:03.178914 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:03.179047 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:03.179242 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 09:14:03.179308 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":252,"threads_started":1}
I20260925 09:14:03.180584 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:03.174107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:45869 (local address 127.3.19.65:48567)
0925 09:14:03.175033 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:03.175068 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:03.176948 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:03.177708 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:03.177717 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:03.177813 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:03.178521 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:03.178524 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:03.178568 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:03.178654 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:03.179009 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:03.179019 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:03.179027 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 09:14:03.179325 (+ 298us) client_negotiation.cc:240] Negotiation successful
0925 09:14:03.179610 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":320,"threads_started":1}
I20260925 09:14:03.182304 8042 heartbeater.cc:347] Connected to a master server at 127.3.19.126:45869
I20260925 09:14:03.182622 8042 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:03.183213 8042 heartbeater.cc:511] Master 127.3.19.126:45869 requested a full tablet report, sending...
I20260925 09:14:03.185520 7856 ts_manager.cc:194] Registered new tserver with Master: aeb6af4b5f1e4587a0db1f19116bcdec (127.3.19.65:42477)
W20260925 09:14:03.511245 8046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:03.511585 8046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:03.511677 8046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:03.519032 8046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:03.519171 8046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:03.519282 8046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:14:03.528581 8046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:03.529965 8046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:03.531818 8046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:03.545759 8051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:03.545759 8052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:03.546128 8054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:03.546626 8046 server_base.cc:1034] running on GCE node
I20260925 09:14:03.547358 8046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:03.548374 8046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:03.549583 8046 hybrid_clock.cc:648] HybridClock initialized: now 1790327643549530 us; error 48 us; skew 500 ppm
I20260925 09:14:03.549938 8046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:03.552393 8046 webserver.cc:460] Webserver started at http://127.3.19.66:37615/ using document root <none> and password file <none>
I20260925 09:14:03.553138 8046 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:03.553300 8046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:03.553645 8046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:03.556602 8046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "e060116b7ae74d8184300d1093f065d3"
format_stamp: "Formatted at 2026-09-25 09:14:03 on dist-test-slave-l6qh"
server_key: "2f64769a7e3bafb353fe45eccbbb0041"
server_key_iv: "4e059c800f1aa28590f6f3d97825adbc"
server_key_version: "encryptionkey@0"
I20260925 09:14:03.557487 8046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "e060116b7ae74d8184300d1093f065d3"
format_stamp: "Formatted at 2026-09-25 09:14:03 on dist-test-slave-l6qh"
server_key: "2f64769a7e3bafb353fe45eccbbb0041"
server_key_iv: "4e059c800f1aa28590f6f3d97825adbc"
server_key_version: "encryptionkey@0"
I20260925 09:14:03.563872 8046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 09:14:03.567644 8060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:03.569460 8046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:03.569674 8046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "e060116b7ae74d8184300d1093f065d3"
format_stamp: "Formatted at 2026-09-25 09:14:03 on dist-test-slave-l6qh"
server_key: "2f64769a7e3bafb353fe45eccbbb0041"
server_key_iv: "4e059c800f1aa28590f6f3d97825adbc"
server_key_version: "encryptionkey@0"
I20260925 09:14:03.569986 8046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:03.575917 8046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:03.576848 8046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:03.577138 8046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:03.578132 8046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:03.579527 8046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:03.579612 8046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:03.579698 8046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:03.579753 8046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:03.618276 8046 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:38797
I20260925 09:14:03.618305 8172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:38797 every 8 connection(s)
I20260925 09:14:03.619978 8046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:14:03.628319 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:03.622651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:44229 (local address 127.3.19.126:45869)
0925 09:14:03.622833 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:03.622839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:03.624504 (+ 1665us) server_negotiation.cc:407] Connection header received
0925 09:14:03.626227 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:03.626232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:03.626289 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:03.626386 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:03.627684 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:03.627688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:03.627691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:03.627715 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:03.627743 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:03.627754 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:03.627846 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:03.628069 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 09:14:03.628116 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 09:14:03.628389 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 8046
I20260925 09:14:03.628556 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:14:03.629144 3149 external_mini_cluster.cc:1428] Setting key 054e5cb05411859979d46fc6e1912a6b
I20260925 09:14:03.629794 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:03.623259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:45869 (local address 127.3.19.66:44229)
0925 09:14:03.624311 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:03.624362 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:03.625958 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:03.626587 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:03.626595 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:03.626688 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:03.627399 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:03.627401 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:03.627440 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:03.627525 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:03.627838 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:03.627845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:03.627850 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 09:14:03.628163 (+ 313us) client_negotiation.cc:240] Negotiation successful
0925 09:14:03.628429 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":329,"threads_started":1}
I20260925 09:14:03.631251 8173 heartbeater.cc:347] Connected to a master server at 127.3.19.126:45869
I20260925 09:14:03.631603 8173 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:03.632139 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:45869
--builtin_ntp_servers=127.3.19.84:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 09:14:03.632236 8173 heartbeater.cc:511] Master 127.3.19.126:45869 requested a full tablet report, sending...
I20260925 09:14:03.633724 7856 ts_manager.cc:194] Registered new tserver with Master: e060116b7ae74d8184300d1093f065d3 (127.3.19.66:38797)
W20260925 09:14:03.969750 8176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:03.970048 8176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:03.970223 8176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:03.978036 8176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:03.978150 8176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:03.978331 8176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:14:03.987680 8176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:03.989009 8176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:03.990779 8176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:04.004172 8182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:04.004213 8181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:04.004401 8184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:04.005159 8176 server_base.cc:1034] running on GCE node
I20260925 09:14:04.005901 8176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:04.006922 8176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:04.008162 8176 hybrid_clock.cc:648] HybridClock initialized: now 1790327644008099 us; error 60 us; skew 500 ppm
I20260925 09:14:04.008514 8176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:04.010952 8176 webserver.cc:460] Webserver started at http://127.3.19.67:40417/ using document root <none> and password file <none>
I20260925 09:14:04.011682 8176 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:04.011816 8176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:04.012151 8176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:04.015134 8176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "bbf41a7ba40c43138d48bf37cc493907"
format_stamp: "Formatted at 2026-09-25 09:14:04 on dist-test-slave-l6qh"
server_key: "26d004d1ca7778a7add13895bf10d4d3"
server_key_iv: "b8c3bd28f264e1a9470516980e24c0fe"
server_key_version: "encryptionkey@0"
I20260925 09:14:04.015944 8176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "bbf41a7ba40c43138d48bf37cc493907"
format_stamp: "Formatted at 2026-09-25 09:14:04 on dist-test-slave-l6qh"
server_key: "26d004d1ca7778a7add13895bf10d4d3"
server_key_iv: "b8c3bd28f264e1a9470516980e24c0fe"
server_key_version: "encryptionkey@0"
I20260925 09:14:04.021281 8176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 09:14:04.024966 8190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:04.026487 8176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 09:14:04.026685 8176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "bbf41a7ba40c43138d48bf37cc493907"
format_stamp: "Formatted at 2026-09-25 09:14:04 on dist-test-slave-l6qh"
server_key: "26d004d1ca7778a7add13895bf10d4d3"
server_key_iv: "b8c3bd28f264e1a9470516980e24c0fe"
server_key_version: "encryptionkey@0"
I20260925 09:14:04.026952 8176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:04.032830 8176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:04.033771 8176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:04.034058 8176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:04.034984 8176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:04.036453 8176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:04.036578 8176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:04.036723 8176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:04.036813 8176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:04.070606 8176 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:45531
I20260925 09:14:04.070658 8302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:45531 every 8 connection(s)
I20260925 09:14:04.071977 8176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:14:04.072999 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 8176
I20260925 09:14:04.073171 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:14:04.073786 3149 external_mini_cluster.cc:1428] Setting key 0cfa2efbe05d528d87fb12bf953afef9
I20260925 09:14:04.081023 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.074992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:53497 (local address 127.3.19.126:45869)
0925 09:14:04.075208 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:04.075215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:04.076950 (+ 1735us) server_negotiation.cc:407] Connection header received
0925 09:14:04.078711 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:04.078716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:04.078789 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:04.078910 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:04.080237 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:04.080242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:04.080247 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:04.080272 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:04.080302 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:04.080314 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:04.080428 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:04.080674 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 09:14:04.080740 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 09:14:04.082049 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.075706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:45869 (local address 127.3.19.67:53497)
0925 09:14:04.076779 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:04.076828 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:04.078454 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:04.079158 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:04.079167 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:04.079272 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:04.079949 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:04.079952 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:04.079992 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:04.080078 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:04.080413 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:04.080424 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:04.080465 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 09:14:04.080743 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 09:14:04.081003 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":372,"threads_started":1}
I20260925 09:14:04.083384 8303 heartbeater.cc:347] Connected to a master server at 127.3.19.126:45869
I20260925 09:14:04.083702 8303 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:04.084378 8303 heartbeater.cc:511] Master 127.3.19.126:45869 requested a full tablet report, sending...
I20260925 09:14:04.085752 7856 ts_manager.cc:194] Registered new tserver with Master: bbf41a7ba40c43138d48bf37cc493907 (127.3.19.67:45531)
I20260925 09:14:04.089510 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:14:04.097304 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.094458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.3.19.126:45869)
0925 09:14:04.094720 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:04.094738 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:04.095303 (+ 565us) server_negotiation.cc:407] Connection header received
0925 09:14:04.095440 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:04.095444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:04.095502 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:04.095589 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:04.096611 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:04.096623 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:04.096626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:04.096642 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:04.096676 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:04.096688 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:04.096793 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:04.096890 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 09:14:04.096943 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 09:14:04.101671 7856 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:14:04.108568 7856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:14:04.128082 8319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.122643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:38797 (local address 127.0.0.1:49726)
0925 09:14:04.124219 (+ 1576us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:04.124236 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:04.124325 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:04.125302 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:04.125310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:04.125420 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:04.126262 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:04.126268 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:04.126300 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:04.126380 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:04.127304 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:04.127314 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:04.127321 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 09:14:04.127487 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 09:14:04.127578 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"thread_start_us":1626,"threads_started":1}
I20260925 09:14:04.128271 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.122208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36374 (local address 127.3.19.65:42477)
0925 09:14:04.124805 (+ 2597us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:04.124814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:04.124841 (+ 27us) server_negotiation.cc:407] Connection header received
0925 09:14:04.124907 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:04.124926 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:04.125135 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:04.125314 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:04.127292 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:04.127301 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:04.127314 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:04.127332 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:04.127390 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:04.127418 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:04.127685 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:04.127811 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 09:14:04.127919 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2451,"thread_start_us":2351,"threads_started":1}
I20260925 09:14:04.128641 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.121746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.3.19.66:38797)
0925 09:14:04.122380 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:04.122394 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:04.124466 (+ 2072us) server_negotiation.cc:407] Connection header received
0925 09:14:04.124554 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:04.124565 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:04.124826 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:04.124994 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:04.126649 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:04.126658 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:04.126673 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:04.126703 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:04.127053 (+ 350us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:04.127078 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:04.127254 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:04.128185 (+ 931us) server_negotiation.cc:299] Negotiation successful
0925 09:14:04.128281 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":335,"threads_started":1}
I20260925 09:14:04.128553 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.121601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:45531 (local address 127.0.0.1:45554)
0925 09:14:04.123052 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:04.123089 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:04.123225 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:04.125817 (+ 2592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:04.125822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:04.125845 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:04.126573 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:04.126575 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:04.126586 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:04.126616 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:04.127937 (+ 1321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:04.127950 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:04.127956 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 09:14:04.128094 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 09:14:04.128145 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1191,"thread_start_us":1186,"threads_started":1}
I20260925 09:14:04.128918 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.122475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:42477 (local address 127.0.0.1:36374)
0925 09:14:04.123685 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:04.123701 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:04.123783 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:04.125847 (+ 2064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:04.125850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:04.125866 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:04.127063 (+ 1197us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:04.127066 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:04.127080 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:04.127122 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:04.127668 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:04.127675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:04.127680 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 09:14:04.127849 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 09:14:04.127909 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"thread_start_us":1521,"threads_started":1}
I20260925 09:14:04.129699 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.122246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.3.19.67:45531)
0925 09:14:04.125317 (+ 3071us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:04.125325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:04.125355 (+ 30us) server_negotiation.cc:407] Connection header received
0925 09:14:04.125415 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:04.125422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:04.125618 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:04.126344 (+ 726us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:04.127690 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:04.127698 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:04.127711 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:04.127732 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:04.127787 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:04.127810 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:04.128750 (+ 940us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:04.128816 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 09:14:04.129341 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2899,"thread_start_us":2734,"threads_started":1}
I20260925 09:14:04.132944 8238 tablet_service.cc:1467] Processing CreateTablet for tablet 7f801269c1f341a1ad0baab1f36763fb (DEFAULT_TABLE table=test-table [id=29a9651a53d14b209608912d203ea09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:04.132863 8108 tablet_service.cc:1467] Processing CreateTablet for tablet 7f801269c1f341a1ad0baab1f36763fb (DEFAULT_TABLE table=test-table [id=29a9651a53d14b209608912d203ea09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:04.132864 7977 tablet_service.cc:1467] Processing CreateTablet for tablet 7f801269c1f341a1ad0baab1f36763fb (DEFAULT_TABLE table=test-table [id=29a9651a53d14b209608912d203ea09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:04.134189 8238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f801269c1f341a1ad0baab1f36763fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:04.134246 7977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f801269c1f341a1ad0baab1f36763fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:04.134248 8108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f801269c1f341a1ad0baab1f36763fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:04.148180 8321 tablet_bootstrap.cc:492] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907: Bootstrap starting.
I20260925 09:14:04.151711 8321 tablet_bootstrap.cc:654] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:04.152654 8322 tablet_bootstrap.cc:492] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3: Bootstrap starting.
I20260925 09:14:04.153139 8321 log.cc:824] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:04.153649 8323 tablet_bootstrap.cc:492] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec: Bootstrap starting.
I20260925 09:14:04.156594 8321 tablet_bootstrap.cc:492] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907: No bootstrap required, opened a new log
I20260925 09:14:04.156904 8321 ts_tablet_manager.cc:1397] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 09:14:04.157519 8322 tablet_bootstrap.cc:654] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:04.157806 8323 tablet_bootstrap.cc:654] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:04.158833 8322 log.cc:824] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:04.158905 8323 log.cc:824] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:04.161823 8322 tablet_bootstrap.cc:492] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3: No bootstrap required, opened a new log
I20260925 09:14:04.162115 8322 ts_tablet_manager.cc:1397] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 09:14:04.162159 8323 tablet_bootstrap.cc:492] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec: No bootstrap required, opened a new log
I20260925 09:14:04.162396 8323 ts_tablet_manager.cc:1397] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 09:14:04.166927 8321 raft_consensus.cc:348] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e060116b7ae74d8184300d1093f065d3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38797 } } peers { permanent_uuid: "bbf41a7ba40c43138d48bf37cc493907" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45531 } } peers { permanent_uuid: "aeb6af4b5f1e4587a0db1f19116bcdec" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 42477 } }
I20260925 09:14:04.167433 8321 raft_consensus.cc:374] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:04.167572 8321 raft_consensus.cc:729] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf41a7ba40c43138d48bf37cc493907, State: Initialized, Role: FOLLOWER
I20260925 09:14:04.168246 8321 consensus_queue.cc:260] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e060116b7ae74d8184300d1093f065d3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38797 } } peers { permanent_uuid: "bbf41a7ba40c43138d48bf37cc493907" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45531 } } peers { permanent_uuid: "aeb6af4b5f1e4587a0db1f19116bcdec" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 42477 } }
I20260925 09:14:04.170449 8322 raft_consensus.cc:348] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e060116b7ae74d8184300d1093f065d3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38797 } } peers { permanent_uuid: "bbf41a7ba40c43138d48bf37cc493907" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45531 } } peers { permanent_uuid: "aeb6af4b5f1e4587a0db1f19116bcdec" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 42477 } }
I20260925 09:14:04.171042 8303 heartbeater.cc:503] Master 127.3.19.126:45869 was elected leader, sending a full tablet report...
I20260925 09:14:04.171712 8321 ts_tablet_manager.cc:1428] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260925 09:14:04.171950 8322 raft_consensus.cc:374] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:04.172065 8322 raft_consensus.cc:729] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e060116b7ae74d8184300d1093f065d3, State: Initialized, Role: FOLLOWER
I20260925 09:14:04.172713 8322 consensus_queue.cc:260] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e060116b7ae74d8184300d1093f065d3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38797 } } peers { permanent_uuid: "bbf41a7ba40c43138d48bf37cc493907" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45531 } } peers { permanent_uuid: "aeb6af4b5f1e4587a0db1f19116bcdec" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 42477 } }
I20260925 09:14:04.177302 8323 raft_consensus.cc:348] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e060116b7ae74d8184300d1093f065d3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38797 } } peers { permanent_uuid: "bbf41a7ba40c43138d48bf37cc493907" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45531 } } peers { permanent_uuid: "aeb6af4b5f1e4587a0db1f19116bcdec" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 42477 } }
I20260925 09:14:04.178022 8323 raft_consensus.cc:374] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:04.178203 8323 raft_consensus.cc:729] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb6af4b5f1e4587a0db1f19116bcdec, State: Initialized, Role: FOLLOWER
I20260925 09:14:04.179378 8323 consensus_queue.cc:260] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e060116b7ae74d8184300d1093f065d3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38797 } } peers { permanent_uuid: "bbf41a7ba40c43138d48bf37cc493907" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45531 } } peers { permanent_uuid: "aeb6af4b5f1e4587a0db1f19116bcdec" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 42477 } }
I20260925 09:14:04.180735 8042 heartbeater.cc:503] Master 127.3.19.126:45869 was elected leader, sending a full tablet report...
I20260925 09:14:04.182289 8323 ts_tablet_manager.cc:1428] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec: Time spent starting tablet: real 0.020s user 0.011s sys 0.003s
I20260925 09:14:04.182981 8173 heartbeater.cc:503] Master 127.3.19.126:45869 was elected leader, sending a full tablet report...
I20260925 09:14:04.183632 8322 ts_tablet_manager.cc:1428] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3: Time spent starting tablet: real 0.021s user 0.011s sys 0.008s
I20260925 09:14:04.213404 8328 raft_consensus.cc:482] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:14:04.213739 8328 raft_consensus.cc:504] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e060116b7ae74d8184300d1093f065d3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38797 } } peers { permanent_uuid: "bbf41a7ba40c43138d48bf37cc493907" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45531 } } peers { permanent_uuid: "aeb6af4b5f1e4587a0db1f19116bcdec" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 42477 } }
I20260925 09:14:04.215399 8328 leader_election.cc:288] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf41a7ba40c43138d48bf37cc493907 (127.3.19.67:45531), aeb6af4b5f1e4587a0db1f19116bcdec (127.3.19.65:42477)
I20260925 09:14:04.219059 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.215912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:53361 (local address 127.3.19.67:45531)
0925 09:14:04.216108 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:04.216117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:04.216884 (+ 767us) server_negotiation.cc:407] Connection header received
0925 09:14:04.217048 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:04.217054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:04.217118 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:04.217248 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:04.218413 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:04.218418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:04.218423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:04.218444 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:04.218481 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:04.218496 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:04.218621 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:04.218728 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 09:14:04.218787 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 09:14:04.219056 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.215835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:48171 (local address 127.3.19.65:42477)
0925 09:14:04.216017 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:04.216023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:04.216884 (+ 861us) server_negotiation.cc:407] Connection header received
0925 09:14:04.217048 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:04.217054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:04.217118 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:04.217248 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:04.218418 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:04.218424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:04.218428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:04.218445 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:04.218481 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:04.218496 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:04.218617 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:04.218729 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 09:14:04.218787 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 09:14:04.219408 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.215737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:45531 (local address 127.3.19.66:53361)
0925 09:14:04.216715 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:04.216740 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:04.216924 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:04.217291 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:04.217296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:04.217320 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:04.218223 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:04.218226 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:04.218237 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:04.218261 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:04.218603 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:04.218608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:04.218613 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 09:14:04.218786 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 09:14:04.218843 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"spinlock_wait_cycles":38784,"thread_start_us":468,"threads_started":1}
I20260925 09:14:04.219496 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.215807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:42477 (local address 127.3.19.66:48171)
0925 09:14:04.216775 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:04.216791 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:04.216887 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:04.217292 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:04.217296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:04.217320 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:04.218170 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:04.218176 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:04.218201 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:04.218251 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:04.218603 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:04.218608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:04.218613 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 09:14:04.218780 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 09:14:04.218855 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":20,"thread_start_us":571,"threads_started":1}
I20260925 09:14:04.220360 8258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f801269c1f341a1ad0baab1f36763fb" candidate_uuid: "e060116b7ae74d8184300d1093f065d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf41a7ba40c43138d48bf37cc493907" is_pre_election: true
I20260925 09:14:04.220360 7997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f801269c1f341a1ad0baab1f36763fb" candidate_uuid: "e060116b7ae74d8184300d1093f065d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb6af4b5f1e4587a0db1f19116bcdec" is_pre_election: true
I20260925 09:14:04.220851 8258 raft_consensus.cc:2454] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e060116b7ae74d8184300d1093f065d3 in term 0.
I20260925 09:14:04.220873 7997 raft_consensus.cc:2454] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e060116b7ae74d8184300d1093f065d3 in term 0.
I20260925 09:14:04.221537 8062 leader_election.cc:302] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb6af4b5f1e4587a0db1f19116bcdec, e060116b7ae74d8184300d1093f065d3; no voters:
I20260925 09:14:04.221935 8328 raft_consensus.cc:2781] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:14:04.222074 8328 raft_consensus.cc:482] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:14:04.222191 8328 raft_consensus.cc:3037] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:04.224638 8328 raft_consensus.cc:504] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e060116b7ae74d8184300d1093f065d3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38797 } } peers { permanent_uuid: "bbf41a7ba40c43138d48bf37cc493907" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45531 } } peers { permanent_uuid: "aeb6af4b5f1e4587a0db1f19116bcdec" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 42477 } }
I20260925 09:14:04.225376 8328 leader_election.cc:288] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [CANDIDATE]: Term 1 election: Requested vote from peers bbf41a7ba40c43138d48bf37cc493907 (127.3.19.67:45531), aeb6af4b5f1e4587a0db1f19116bcdec (127.3.19.65:42477)
I20260925 09:14:04.225796 8258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f801269c1f341a1ad0baab1f36763fb" candidate_uuid: "e060116b7ae74d8184300d1093f065d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf41a7ba40c43138d48bf37cc493907"
I20260925 09:14:04.225903 7997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f801269c1f341a1ad0baab1f36763fb" candidate_uuid: "e060116b7ae74d8184300d1093f065d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb6af4b5f1e4587a0db1f19116bcdec"
I20260925 09:14:04.226008 8258 raft_consensus.cc:3037] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:04.226162 7997 raft_consensus.cc:3037] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:04.228322 8258 raft_consensus.cc:2454] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e060116b7ae74d8184300d1093f065d3 in term 1.
I20260925 09:14:04.228374 7997 raft_consensus.cc:2454] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e060116b7ae74d8184300d1093f065d3 in term 1.
I20260925 09:14:04.228755 8061 leader_election.cc:302] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbf41a7ba40c43138d48bf37cc493907, e060116b7ae74d8184300d1093f065d3; no voters:
I20260925 09:14:04.229089 8328 raft_consensus.cc:2781] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:04.229611 8328 raft_consensus.cc:686] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [term 1 LEADER]: Becoming Leader. State: Replica: e060116b7ae74d8184300d1093f065d3, State: Running, Role: LEADER
I20260925 09:14:04.230176 8328 consensus_queue.cc:237] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e060116b7ae74d8184300d1093f065d3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38797 } } peers { permanent_uuid: "bbf41a7ba40c43138d48bf37cc493907" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45531 } } peers { permanent_uuid: "aeb6af4b5f1e4587a0db1f19116bcdec" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 42477 } }
I20260925 09:14:04.234835 7855 catalog_manager.cc:5495] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e060116b7ae74d8184300d1093f065d3 (127.3.19.66). New cstate: current_term: 1 leader_uuid: "e060116b7ae74d8184300d1093f065d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e060116b7ae74d8184300d1093f065d3" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbf41a7ba40c43138d48bf37cc493907" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb6af4b5f1e4587a0db1f19116bcdec" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 42477 } health_report { overall_health: UNKNOWN } } }
I20260925 09:14:04.256891 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.254910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.3.19.66:38797)
0925 09:14:04.255088 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:04.255093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:04.255149 (+ 56us) server_negotiation.cc:407] Connection header received
0925 09:14:04.255355 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:04.255361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:04.255456 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:04.255567 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:04.256377 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:04.256382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:04.256385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:04.256400 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:04.256431 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:04.256442 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:04.256546 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:04.256614 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 09:14:04.256658 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 09:14:04.265329 7997 raft_consensus.cc:1261] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec [term 1 FOLLOWER]: Refusing update from remote peer e060116b7ae74d8184300d1093f065d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:04.265333 8258 raft_consensus.cc:1261] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907 [term 1 FOLLOWER]: Refusing update from remote peer e060116b7ae74d8184300d1093f065d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:04.266263 8332 consensus_queue.cc:1035] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf41a7ba40c43138d48bf37cc493907" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:04.266649 8328 consensus_queue.cc:1035] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb6af4b5f1e4587a0db1f19116bcdec" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 42477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:04.275866 8335 mvcc.cc:205] Tried to move back new op lower bound from 7333182030900592640 to 7333182030771453952. Current Snapshot: MvccSnapshot[applied={T|T < 7333182030900592640}]
I20260925 09:14:04.280349 8337 mvcc.cc:205] Tried to move back new op lower bound from 7333182030900592640 to 7333182030771453952. Current Snapshot: MvccSnapshot[applied={T|T < 7333182030900592640}]
I20260925 09:14:04.291342 7856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:04.291622 7856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:04.295429 7856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fe86b5cd5f84f1baeb4898b7d2fe5c9: Sending DeleteTablet for 3 replicas of tablet 7f801269c1f341a1ad0baab1f36763fb
I20260925 09:14:04.296613 7977 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f801269c1f341a1ad0baab1f36763fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:04 UTC) from {username='slave'} at 127.0.0.1:36374
I20260925 09:14:04.296623 8108 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f801269c1f341a1ad0baab1f36763fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:04 UTC) from {username='slave'} at 127.0.0.1:49726
I20260925 09:14:04.296656 8238 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f801269c1f341a1ad0baab1f36763fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:04 UTC) from {username='slave'} at 127.0.0.1:45554
I20260925 09:14:04.300171 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 7915
I20260925 09:14:04.312238 8347 tablet_replica.cc:330] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907: stopping tablet replica
I20260925 09:14:04.312865 8347 raft_consensus.cc:2229] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:04.313493 8347 raft_consensus.cc:2258] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:04.315816 8347 ts_tablet_manager.cc:1905] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:04.323030 8347 ts_tablet_manager.cc:1918] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:04.323256 8347 log.cc:1196] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/7f801269c1f341a1ad0baab1f36763fb
I20260925 09:14:04.323889 8347 ts_tablet_manager.cc:1939] T 7f801269c1f341a1ad0baab1f36763fb P bbf41a7ba40c43138d48bf37cc493907: Deleting consensus metadata
I20260925 09:14:04.326020 7840 catalog_manager.cc:4841] TS bbf41a7ba40c43138d48bf37cc493907 (127.3.19.67:45531): tablet 7f801269c1f341a1ad0baab1f36763fb (table test-table [id=29a9651a53d14b209608912d203ea09f]) successfully deleted
I20260925 09:14:04.329769 8345 tablet_replica.cc:330] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec: stopping tablet replica
I20260925 09:14:04.330384 8345 raft_consensus.cc:2229] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:04.330947 8346 tablet_replica.cc:330] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3: stopping tablet replica
I20260925 09:14:04.331008 8345 raft_consensus.cc:2258] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:04.331508 8346 raft_consensus.cc:2229] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:14:04.332119 8346 raft_consensus.cc:2258] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:04.333169 8345 ts_tablet_manager.cc:1905] T 7f801269c1f341a1ad0baab1f36763fb P aeb6af4b5f1e4587a0db1f19116bcdec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:04.333856 8346 ts_tablet_manager.cc:1905] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:04.336102 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.301889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.3.19.65:42477)
0925 09:14:04.302111 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:04.302120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:04.328027 (+ 25907us) server_negotiation.cc:407] Connection header received
0925 09:14:04.328620 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:04.328630 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:04.328771 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:04.328862 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:04.335049 (+ 6187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:04.335057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:04.335064 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:04.335088 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:04.335130 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:04.335145 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:04.335266 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:04.335709 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 09:14:04.335773 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 09:14:04.336804 8037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:04.343272 8346 ts_tablet_manager.cc:1918] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:04.343487 8346 log.cc:1196] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/7f801269c1f341a1ad0baab1f36763fb
I20260925 09:14:04.343987 8346 ts_tablet_manager.cc:1939] T 7f801269c1f341a1ad0baab1f36763fb P e060116b7ae74d8184300d1093f065d3: Deleting consensus metadata
I20260925 09:14:04.345794 7842 catalog_manager.cc:4841] TS e060116b7ae74d8184300d1093f065d3 (127.3.19.66:38797): tablet 7f801269c1f341a1ad0baab1f36763fb (table test-table [id=29a9651a53d14b209608912d203ea09f]) successfully deleted
I20260925 09:14:04.449541 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 7915
W20260925 09:14:04.473480 7841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.19.65:42477 (error 108)
I20260925 09:14:04.474267 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 8046
I20260925 09:14:04.475800 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.475244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:42477 (local address 127.0.0.1:36392)
0925 09:14:04.475493 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:04.475625 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260925 09:14:04.476056 7841 catalog_manager.cc:4584] TS aeb6af4b5f1e4587a0db1f19116bcdec (127.3.19.65:42477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f801269c1f341a1ad0baab1f36763fb: Network error: Client connection negotiation failed: client connection to 127.3.19.65:42477: connect: Connection refused (error 111)
I20260925 09:14:04.477596 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.475140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49744 (local address 127.3.19.66:38797)
0925 09:14:04.475319 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:04.475325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:04.475640 (+ 315us) server_negotiation.cc:407] Connection header received
0925 09:14:04.475747 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:04.475753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:04.475813 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:04.475917 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:04.476941 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:04.476945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:04.476948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:04.476969 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:04.476996 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:04.477008 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:04.477118 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:04.477305 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 09:14:04.477366 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 09:14:04.478302 8168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:04.529702 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.529084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:42477 (local address 127.0.0.1:36408)
0925 09:14:04.529376 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:04.529511 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260925 09:14:04.530833 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.530444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:42477 (local address 127.0.0.1:36418)
0925 09:14:04.530610 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:04.530697 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 09:14:04.589573 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 8046
I20260925 09:14:04.591629 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.591094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:42477 (local address 127.0.0.1:36424)
0925 09:14:04.591365 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:04.591454 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 09:14:04.592571 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.592237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:42477 (local address 127.0.0.1:36434)
0925 09:14:04.592375 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:04.592457 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 09:14:04.613807 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 8176
I20260925 09:14:04.616685 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.614552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.3.19.67:45531)
0925 09:14:04.614779 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:04.614785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:04.614804 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:14:04.614909 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:04.614915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:04.614997 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:04.615104 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:04.616035 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:04.616041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:04.616048 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:04.616070 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:04.616100 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:04.616113 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:04.616251 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:04.616347 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 09:14:04.616406 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260925 09:14:04.617374 8298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:04.673010 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.672524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:42477 (local address 127.0.0.1:36450)
0925 09:14:04.672765 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:04.672862 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 09:14:04.674196 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:04.673800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:42477 (local address 127.0.0.1:36454)
0925 09:14:04.674012 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:04.674086 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 09:14:04.705996 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 8176
I20260925 09:14:04.722815 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 7825
I20260925 09:14:04.723999 7886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:04.829217 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 7825
2026-09-25T09:14:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2716 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 09:14:04.871229 3149 mini_oidc.cc:152] Starting JWKS server
I20260925 09:14:04.873762 3149 webserver.cc:460] Webserver started at https://127.0.0.1:37009/ using document root <none> and password file <none>
I20260925 09:14:04.874094 3149 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 09:14:04.874735 3149 webserver.cc:460] Webserver started at http://127.0.0.1:38349/ using document root <none> and password file <none>
2026-09-25T09:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:14:04Z Disabled control of system clock
I20260925 09:14:04.903156 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:37019
--webserver_interface=127.3.19.126
--webserver_port=0
--builtin_ntp_servers=127.3.19.84:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:37019
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37009/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/testchainca.cert with env {}
W20260925 09:14:05.246465 8359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:05.246798 8359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:05.246949 8359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:05.254460 8359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:05.254580 8359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:05.254674 8359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:05.254783 8359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:05.254866 8359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 09:14:05.254969 8359 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 09:14:05.255028 8359 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37009/jwks.json
I20260925 09:14:05.265296 8359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:37019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:37019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37009/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:05.266678 8359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:05.268034 8359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:05.280002 8365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:05.280063 8364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:05.279973 8367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:05.281028 8359 server_base.cc:1034] running on GCE node
I20260925 09:14:05.281955 8359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:05.283452 8359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:05.284695 8359 hybrid_clock.cc:648] HybridClock initialized: now 1790327645284650 us; error 30 us; skew 500 ppm
I20260925 09:14:05.285118 8359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:05.287560 8359 webserver.cc:460] Webserver started at http://127.3.19.126:44315/ using document root <none> and password file <none>
I20260925 09:14:05.288309 8359 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:05.288443 8359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:05.288767 8359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:05.291929 8359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "622a06f425f14e0bb51605c1fdfed573"
format_stamp: "Formatted at 2026-09-25 09:14:05 on dist-test-slave-l6qh"
server_key: "b5b69fefa5f8a9b4eb448734261c8e81"
server_key_iv: "453de13cdbc3328103e6c3d64da4be58"
server_key_version: "encryptionkey@0"
I20260925 09:14:05.292761 8359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "622a06f425f14e0bb51605c1fdfed573"
format_stamp: "Formatted at 2026-09-25 09:14:05 on dist-test-slave-l6qh"
server_key: "b5b69fefa5f8a9b4eb448734261c8e81"
server_key_iv: "453de13cdbc3328103e6c3d64da4be58"
server_key_version: "encryptionkey@0"
I20260925 09:14:05.298138 8359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 09:14:05.301723 8373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:05.303059 8359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 09:14:05.303254 8359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "622a06f425f14e0bb51605c1fdfed573"
format_stamp: "Formatted at 2026-09-25 09:14:05 on dist-test-slave-l6qh"
server_key: "b5b69fefa5f8a9b4eb448734261c8e81"
server_key_iv: "453de13cdbc3328103e6c3d64da4be58"
server_key_version: "encryptionkey@0"
I20260925 09:14:05.303493 8359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:05.339660 8359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:05.340570 8359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:05.340873 8359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:05.360046 8359 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:37019
I20260925 09:14:05.360065 8424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:37019 every 8 connection(s)
I20260925 09:14:05.361496 8359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:14:05.364526 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 8359
I20260925 09:14:05.364744 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:14:05.365554 3149 external_mini_cluster.cc:1428] Setting key 9f9cb5c58fd2839ec16ead1e0c36a4ab
I20260925 09:14:05.365904 8425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:05.381210 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:05.367617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.3.19.126:37019)
0925 09:14:05.368829 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:05.368840 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:05.368886 (+ 46us) server_negotiation.cc:407] Connection header received
0925 09:14:05.370952 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:05.370975 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:05.371483 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:05.371832 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:05.372533 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:05.373988 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:05.375878 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:05.376639 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:05.376786 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:05.378227 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:05.378242 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:05.378262 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:05.379036 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:05.379181 (+ 145us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:05.379207 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:05.379320 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:05.379543 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 09:14:05.379875 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":391,"threads_started":1}
I20260925 09:14:05.381443 8425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573: Bootstrap starting.
I20260925 09:14:05.385538 8425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:05.385836 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:14:05.386862 8425 log.cc:824] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:05.390111 8425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573: No bootstrap required, opened a new log
I20260925 09:14:05.397418 8425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622a06f425f14e0bb51605c1fdfed573" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37019 } }
I20260925 09:14:05.397804 8425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:05.397884 8425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622a06f425f14e0bb51605c1fdfed573, State: Initialized, Role: FOLLOWER
I20260925 09:14:05.398370 8425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622a06f425f14e0bb51605c1fdfed573" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37019 } }
I20260925 09:14:05.398559 8425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:14:05.398648 8425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:14:05.398772 8425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:05.401073 8425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622a06f425f14e0bb51605c1fdfed573" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37019 } }
I20260925 09:14:05.401643 8425 leader_election.cc:302] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 622a06f425f14e0bb51605c1fdfed573; no voters:
I20260925 09:14:05.402170 8425 leader_election.cc:288] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:14:05.402315 8430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:05.403695 8430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [term 1 LEADER]: Becoming Leader. State: Replica: 622a06f425f14e0bb51605c1fdfed573, State: Running, Role: LEADER
I20260925 09:14:05.404261 8430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622a06f425f14e0bb51605c1fdfed573" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37019 } }
I20260925 09:14:05.404415 8425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:05.407440 8431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "622a06f425f14e0bb51605c1fdfed573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622a06f425f14e0bb51605c1fdfed573" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37019 } } }
I20260925 09:14:05.407513 8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 622a06f425f14e0bb51605c1fdfed573. Latest consensus state: current_term: 1 leader_uuid: "622a06f425f14e0bb51605c1fdfed573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622a06f425f14e0bb51605c1fdfed573" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37019 } } }
I20260925 09:14:05.407917 8431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:05.407931 8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:05.414330 8438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:14:05.419571 8438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:14:05.428678 8438 catalog_manager.cc:1347] Generated new cluster ID: a7358a69ceb94de294c0c8f0f85de6e7
I20260925 09:14:05.428808 8438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:14:05.440682 8438 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:14:05.442603 8438 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:14:05.455915 8438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 622a06f425f14e0bb51605c1fdfed573: Generated new TSK 0
I20260925 09:14:05.457046 8438 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 09:14:05.500928 8455 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 09:14:05.501116 8455 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 09:14:05.502247 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:05.481563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.3.19.126:37019)
0925 09:14:05.481778 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:05.481790 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:05.482216 (+ 426us) server_negotiation.cc:407] Connection header received
0925 09:14:05.482563 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:05.482567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:05.482639 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:05.482793 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0925 09:14:05.483341 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:05.484070 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:05.485568 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:05.486019 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:05.486158 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:05.486715 (+ 557us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 09:14:05.501533 (+ 14818us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37009/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 09:14:05.501918 (+ 385us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37009/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 09:14:05.502374 8428 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37009/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 09:14:05.503648 8454 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:05.481450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:37019 (local address 127.0.0.1:51656)
0925 09:14:05.482102 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:05.482117 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:05.482332 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:05.482815 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:05.482820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:05.482832 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0925 09:14:05.483174 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:05.483182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:05.484215 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:05.484220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:05.485415 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:05.485426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:05.486224 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:05.486231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:05.486364 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:05.486420 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 09:14:05.501829 (+ 15409us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37009/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 09:14:05.502190 (+ 361us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.19.126:37019: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37009/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":329,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.19.126:37019: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37009/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 09:14:05.505913 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 8359
I20260925 09:14:05.507143 8420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:05.604041 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 8359
2026-09-25T09:14:05Z chronyd exiting
I20260925 09:14:05.655676 3149 test_util.cc:175] -----------------------------------------------
I20260925 09:14:05.655923 3149 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790327600440376-3149-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (789 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 09:14:05.661479 3149 mini_oidc.cc:152] Starting JWKS server
I20260925 09:14:05.663430 3149 webserver.cc:460] Webserver started at https://127.0.0.1:33119/ using document root <none> and password file <none>
I20260925 09:14:05.663697 3149 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 09:14:05.664381 3149 webserver.cc:460] Webserver started at http://127.0.0.1:38893/ using document root <none> and password file <none>
2026-09-25T09:14:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:14:05Z Disabled control of system clock
I20260925 09:14:05.694038 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:40999
--webserver_interface=127.3.19.126
--webserver_port=0
--builtin_ntp_servers=127.3.19.84:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:40999
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33119/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/testchainca.cert with env {}
W20260925 09:14:06.024161 8462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:06.024474 8462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:06.024641 8462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:06.031965 8462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:06.032094 8462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:06.032131 8462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:06.032261 8462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:06.032338 8462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 09:14:06.032456 8462 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 09:14:06.032536 8462 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33119/jwks.json
I20260925 09:14:06.042914 8462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:40999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:40999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33119/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:06.044332 8462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:06.045704 8462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:06.057641 8468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:06.057816 8467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:06.057751 8470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:06.058871 8462 server_base.cc:1034] running on GCE node
I20260925 09:14:06.059703 8462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:06.061304 8462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:06.062573 8462 hybrid_clock.cc:648] HybridClock initialized: now 1790327646062522 us; error 47 us; skew 500 ppm
I20260925 09:14:06.062976 8462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:06.065498 8462 webserver.cc:460] Webserver started at http://127.3.19.126:44615/ using document root <none> and password file <none>
I20260925 09:14:06.066233 8462 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:06.066378 8462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:06.066717 8462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:06.069783 8462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "b20ebae29b9a4d16a94889303e22845f"
format_stamp: "Formatted at 2026-09-25 09:14:06 on dist-test-slave-l6qh"
server_key: "808c095eb21e7ab3525346ef462c237a"
server_key_iv: "9576557676866e9df68a9b0c37211d23"
server_key_version: "encryptionkey@0"
I20260925 09:14:06.070604 8462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "b20ebae29b9a4d16a94889303e22845f"
format_stamp: "Formatted at 2026-09-25 09:14:06 on dist-test-slave-l6qh"
server_key: "808c095eb21e7ab3525346ef462c237a"
server_key_iv: "9576557676866e9df68a9b0c37211d23"
server_key_version: "encryptionkey@0"
I20260925 09:14:06.075870 8462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:14:06.079394 8476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:06.080849 8462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:14:06.081058 8462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "b20ebae29b9a4d16a94889303e22845f"
format_stamp: "Formatted at 2026-09-25 09:14:06 on dist-test-slave-l6qh"
server_key: "808c095eb21e7ab3525346ef462c237a"
server_key_iv: "9576557676866e9df68a9b0c37211d23"
server_key_version: "encryptionkey@0"
I20260925 09:14:06.081323 8462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:06.103596 8462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:06.104497 8462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:06.104805 8462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:06.123729 8462 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:40999
I20260925 09:14:06.123749 8527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:40999 every 8 connection(s)
I20260925 09:14:06.125171 8462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:14:06.129015 8528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:06.135088 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 8462
I20260925 09:14:06.135320 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:14:06.135975 3149 external_mini_cluster.cc:1428] Setting key aaa623749834509978796cc56c060950
I20260925 09:14:06.140394 8528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f: Bootstrap starting.
I20260925 09:14:06.143695 8528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:06.144757 8528 log.cc:824] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:06.147946 8528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f: No bootstrap required, opened a new log
I20260925 09:14:06.150144 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:06.138028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.3.19.126:40999)
0925 09:14:06.139052 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:06.139063 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:06.139121 (+ 58us) server_negotiation.cc:407] Connection header received
0925 09:14:06.140885 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:06.140909 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:06.141457 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:06.141867 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:06.142445 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:06.143731 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:06.145148 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:06.146336 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:06.146544 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:06.147748 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:06.147756 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:06.147771 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:06.148367 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:06.148453 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:06.148483 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:06.148619 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:06.148828 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 09:14:06.149131 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":354,"threads_started":1}
I20260925 09:14:06.154196 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:14:06.156646 8528 raft_consensus.cc:348] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20ebae29b9a4d16a94889303e22845f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 40999 } }
I20260925 09:14:06.157070 8528 raft_consensus.cc:374] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:06.157195 8528 raft_consensus.cc:729] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20ebae29b9a4d16a94889303e22845f, State: Initialized, Role: FOLLOWER
I20260925 09:14:06.157862 8528 consensus_queue.cc:260] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20ebae29b9a4d16a94889303e22845f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 40999 } }
I20260925 09:14:06.158118 8528 raft_consensus.cc:388] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:14:06.158293 8528 raft_consensus.cc:482] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:14:06.158479 8528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:06.161002 8528 raft_consensus.cc:504] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20ebae29b9a4d16a94889303e22845f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 40999 } }
I20260925 09:14:06.161593 8528 leader_election.cc:302] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b20ebae29b9a4d16a94889303e22845f; no voters:
I20260925 09:14:06.162171 8528 leader_election.cc:288] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:14:06.162335 8533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:06.163043 8533 raft_consensus.cc:686] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [term 1 LEADER]: Becoming Leader. State: Replica: b20ebae29b9a4d16a94889303e22845f, State: Running, Role: LEADER
I20260925 09:14:06.163566 8533 consensus_queue.cc:237] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20ebae29b9a4d16a94889303e22845f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 40999 } }
I20260925 09:14:06.164311 8528 sys_catalog.cc:564] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:06.166635 8535 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b20ebae29b9a4d16a94889303e22845f. Latest consensus state: current_term: 1 leader_uuid: "b20ebae29b9a4d16a94889303e22845f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20ebae29b9a4d16a94889303e22845f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 40999 } } }
I20260925 09:14:06.167109 8535 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:06.167338 8534 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b20ebae29b9a4d16a94889303e22845f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20ebae29b9a4d16a94889303e22845f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 40999 } } }
I20260925 09:14:06.168620 8534 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:06.169783 8542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:14:06.174331 8542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:14:06.182904 8542 catalog_manager.cc:1347] Generated new cluster ID: 217743d588774697892abc22384bb1c1
I20260925 09:14:06.183044 8542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:14:06.205564 8542 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:14:06.206861 8542 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:14:06.221426 8542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b20ebae29b9a4d16a94889303e22845f: Generated new TSK 0
I20260925 09:14:06.222419 8542 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 09:14:06.266600 8558 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 09:14:06.266744 8558 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 09:14:06.267936 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:06.247114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.3.19.126:40999)
0925 09:14:06.247374 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:06.247383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:06.247739 (+ 356us) server_negotiation.cc:407] Connection header received
0925 09:14:06.248011 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:06.248020 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:06.248103 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:06.248262 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0925 09:14:06.248801 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:06.249666 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:06.251778 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:06.252270 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:06.252378 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:06.253008 (+ 630us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 09:14:06.267276 (+ 14268us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 09:14:06.267673 (+ 397us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57064: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 09:14:06.268097 8531 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57064: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 09:14:06.268711 8557 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:06.246983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:40999 (local address 127.0.0.1:57064)
0925 09:14:06.247616 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:06.247637 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:06.247796 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:06.248301 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:06.248306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:06.248319 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0925 09:14:06.248629 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:06.248641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:06.249818 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:06.249824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:06.251609 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:06.251623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:06.252448 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:06.252454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:06.252652 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:06.252703 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 09:14:06.267539 (+ 14836us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 09:14:06.267854 (+ 315us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.19.126:40999: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":373,"threads_started":1}
I20260925 09:14:06.269861 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 8462
I20260925 09:14:06.271116 8523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:06.365154 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 8462
2026-09-25T09:14:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (761 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 09:14:06.420995 3149 mini_oidc.cc:152] Starting JWKS server
I20260925 09:14:06.422899 3149 webserver.cc:460] Webserver started at https://127.0.0.1:45489/ using document root <none> and password file <none>
I20260925 09:14:06.423166 3149 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 09:14:06.423791 3149 webserver.cc:460] Webserver started at http://127.0.0.1:45067/ using document root <none> and password file <none>
2026-09-25T09:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:14:06Z Disabled control of system clock
I20260925 09:14:06.452528 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:44019
--webserver_interface=127.3.19.126
--webserver_port=0
--builtin_ntp_servers=127.3.19.84:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:44019
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45489/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 09:14:06.790513 8565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:06.790879 8565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:06.791043 8565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:06.798489 8565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:06.798655 8565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:06.798731 8565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:06.798842 8565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:06.798920 8565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 09:14:06.799009 8565 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 09:14:06.799105 8565 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45489/jwks.json
I20260925 09:14:06.808648 8565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:44019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:44019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45489/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:06.810019 8565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:06.811846 8565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:06.824129 8570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:06.824103 8573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:06.824313 8571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:06.824926 8565 server_base.cc:1034] running on GCE node
I20260925 09:14:06.825811 8565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:06.827248 8565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:06.828418 8565 hybrid_clock.cc:648] HybridClock initialized: now 1790327646828310 us; error 96 us; skew 500 ppm
I20260925 09:14:06.828809 8565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:06.831080 8565 webserver.cc:460] Webserver started at http://127.3.19.126:36977/ using document root <none> and password file <none>
I20260925 09:14:06.831750 8565 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:06.831840 8565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:06.832109 8565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:06.835022 8565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "a350d0c11b804b9eb6cfef4eb34beee7"
format_stamp: "Formatted at 2026-09-25 09:14:06 on dist-test-slave-l6qh"
server_key: "d4fea7a9e4bb2f4f7f03f36f43f6ae28"
server_key_iv: "533c176ee38b27b674f60f987d7aa768"
server_key_version: "encryptionkey@0"
I20260925 09:14:06.835798 8565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "a350d0c11b804b9eb6cfef4eb34beee7"
format_stamp: "Formatted at 2026-09-25 09:14:06 on dist-test-slave-l6qh"
server_key: "d4fea7a9e4bb2f4f7f03f36f43f6ae28"
server_key_iv: "533c176ee38b27b674f60f987d7aa768"
server_key_version: "encryptionkey@0"
I20260925 09:14:06.840950 8565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 09:14:06.844393 8579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:06.846199 8565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:06.846413 8565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "a350d0c11b804b9eb6cfef4eb34beee7"
format_stamp: "Formatted at 2026-09-25 09:14:06 on dist-test-slave-l6qh"
server_key: "d4fea7a9e4bb2f4f7f03f36f43f6ae28"
server_key_iv: "533c176ee38b27b674f60f987d7aa768"
server_key_version: "encryptionkey@0"
I20260925 09:14:06.846683 8565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:06.870715 8565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:06.871596 8565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:06.871887 8565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:06.890775 8565 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:44019
I20260925 09:14:06.890794 8630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:44019 every 8 connection(s)
I20260925 09:14:06.892195 8565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:14:06.892807 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 8565
I20260925 09:14:06.893070 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:14:06.893940 3149 external_mini_cluster.cc:1428] Setting key fed48d83ce9105655529d94569dc8402
I20260925 09:14:06.896752 8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:06.907771 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:06.895911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37116 (local address 127.3.19.126:44019)
0925 09:14:06.896985 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:06.896997 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:06.897044 (+ 47us) server_negotiation.cc:407] Connection header received
0925 09:14:06.898655 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:06.898678 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:06.899167 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:06.899608 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:06.900181 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:06.901614 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:06.902943 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:06.903683 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:06.903858 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:06.905203 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:06.905211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:06.905293 (+ 82us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:06.905938 (+ 645us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:06.906035 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:06.906062 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:06.906168 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:06.906436 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 09:14:06.906791 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":352,"threads_started":1}
I20260925 09:14:06.908603 8631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7: Bootstrap starting.
I20260925 09:14:06.912005 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:14:06.912283 8631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:06.913347 8631 log.cc:824] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:06.916369 8631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7: No bootstrap required, opened a new log
I20260925 09:14:06.923743 8631 raft_consensus.cc:348] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a350d0c11b804b9eb6cfef4eb34beee7" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44019 } }
I20260925 09:14:06.924291 8631 raft_consensus.cc:374] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:06.924413 8631 raft_consensus.cc:729] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a350d0c11b804b9eb6cfef4eb34beee7, State: Initialized, Role: FOLLOWER
I20260925 09:14:06.925132 8631 consensus_queue.cc:260] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a350d0c11b804b9eb6cfef4eb34beee7" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44019 } }
I20260925 09:14:06.925414 8631 raft_consensus.cc:388] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:14:06.925576 8631 raft_consensus.cc:482] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:14:06.925765 8631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:06.927898 8631 raft_consensus.cc:504] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a350d0c11b804b9eb6cfef4eb34beee7" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44019 } }
I20260925 09:14:06.928419 8631 leader_election.cc:302] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a350d0c11b804b9eb6cfef4eb34beee7; no voters:
I20260925 09:14:06.929014 8631 leader_election.cc:288] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:14:06.929176 8636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:06.929982 8636 raft_consensus.cc:686] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [term 1 LEADER]: Becoming Leader. State: Replica: a350d0c11b804b9eb6cfef4eb34beee7, State: Running, Role: LEADER
I20260925 09:14:06.930562 8636 consensus_queue.cc:237] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a350d0c11b804b9eb6cfef4eb34beee7" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44019 } }
I20260925 09:14:06.931334 8631 sys_catalog.cc:564] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:06.933775 8638 sys_catalog.cc:455] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a350d0c11b804b9eb6cfef4eb34beee7. Latest consensus state: current_term: 1 leader_uuid: "a350d0c11b804b9eb6cfef4eb34beee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a350d0c11b804b9eb6cfef4eb34beee7" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44019 } } }
I20260925 09:14:06.933734 8637 sys_catalog.cc:455] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a350d0c11b804b9eb6cfef4eb34beee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a350d0c11b804b9eb6cfef4eb34beee7" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 44019 } } }
I20260925 09:14:06.934214 8638 sys_catalog.cc:458] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:06.934329 8637 sys_catalog.cc:458] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:06.935981 8644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:14:06.941201 8644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:14:06.951152 8644 catalog_manager.cc:1347] Generated new cluster ID: 39192b51920d41a9ae2827d1cf3ae3ce
I20260925 09:14:06.951287 8644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:14:06.960995 8644 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:14:06.962391 8644 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:14:06.970191 8644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a350d0c11b804b9eb6cfef4eb34beee7: Generated new TSK 0
I20260925 09:14:06.971096 8644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:14:06.994868 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:06.985948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.3.19.126:44019)
0925 09:14:06.986156 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:06.986163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:06.986574 (+ 411us) server_negotiation.cc:407] Connection header received
0925 09:14:06.986976 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:06.986994 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:06.987113 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:06.987299 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0925 09:14:06.987821 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:06.988553 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:06.990657 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:06.991133 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:06.991235 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:06.991842 (+ 607us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 09:14:06.994253 (+ 2411us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45489/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 09:14:06.994572 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45489/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":78}
W20260925 09:14:06.995002 8634 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45489/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 09:14:06.997043 8660 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:06.985801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:44019 (local address 127.0.0.1:37124)
0925 09:14:06.986442 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:06.986459 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:06.986738 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:06.987334 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:06.987339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:06.987353 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0925 09:14:06.987661 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:06.987673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:06.988703 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:06.988711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:06.990484 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:06.990500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:06.991338 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:06.991345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:06.991532 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:06.991581 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 09:14:06.994489 (+ 2908us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45489/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 09:14:06.996545 (+ 2056us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.19.126:44019: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45489/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":1726,"thread_start_us":348,"threads_started":1}
I20260925 09:14:06.998610 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 8565
I20260925 09:14:07.000162 8626 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 09:14:07.006870 8661 webserver.cc:533] Webserver: sslize failed:
E20260925 09:14:07.006961 8661 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 09:14:07.088963 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 8565
2026-09-25T09:14:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (723 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T09:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:14:07Z Disabled control of system clock
I20260925 09:14:07.174163 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:42801
--webserver_interface=127.3.19.126
--webserver_port=0
--builtin_ntp_servers=127.3.19.84:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:42801
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/testchainca.cert with env {}
W20260925 09:14:07.509663 8666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:07.509981 8666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:07.510164 8666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:07.517652 8666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:07.517769 8666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:07.517843 8666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:07.517961 8666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:07.518040 8666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 09:14:07.518150 8666 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 09:14:07.518232 8666 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 09:14:07.528456 8666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:42801
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:42801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:07.529875 8666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:07.531253 8666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:07.543658 8671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:07.543702 8674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:07.543612 8672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:07.544796 8666 server_base.cc:1034] running on GCE node
I20260925 09:14:07.545734 8666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:07.547248 8666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:07.548455 8666 hybrid_clock.cc:648] HybridClock initialized: now 1790327647548409 us; error 40 us; skew 500 ppm
I20260925 09:14:07.548846 8666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:07.551328 8666 webserver.cc:460] Webserver started at http://127.3.19.126:33765/ using document root <none> and password file <none>
I20260925 09:14:07.552093 8666 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:07.552189 8666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:07.552556 8666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:07.555712 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "375eb9af3f5146438d7e0fd192b61414"
format_stamp: "Formatted at 2026-09-25 09:14:07 on dist-test-slave-l6qh"
server_key: "7c8c28830b607bbc6eacb19e1fa0d98e"
server_key_iv: "aad294fe4831960f6bf0950b124a4469"
server_key_version: "encryptionkey@0"
I20260925 09:14:07.556561 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "375eb9af3f5146438d7e0fd192b61414"
format_stamp: "Formatted at 2026-09-25 09:14:07 on dist-test-slave-l6qh"
server_key: "7c8c28830b607bbc6eacb19e1fa0d98e"
server_key_iv: "aad294fe4831960f6bf0950b124a4469"
server_key_version: "encryptionkey@0"
I20260925 09:14:07.562017 8666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 09:14:07.565837 8680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:07.567378 8666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 09:14:07.567593 8666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "375eb9af3f5146438d7e0fd192b61414"
format_stamp: "Formatted at 2026-09-25 09:14:07 on dist-test-slave-l6qh"
server_key: "7c8c28830b607bbc6eacb19e1fa0d98e"
server_key_iv: "aad294fe4831960f6bf0950b124a4469"
server_key_version: "encryptionkey@0"
I20260925 09:14:07.567836 8666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:07.586424 8666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:07.587321 8666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:07.587617 8666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:07.606629 8666 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:42801
I20260925 09:14:07.606649 8731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:42801 every 8 connection(s)
I20260925 09:14:07.608081 8666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:14:07.611928 8732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:07.615916 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 8666
I20260925 09:14:07.616132 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:14:07.616776 3149 external_mini_cluster.cc:1428] Setting key 56a602a9214a519644869bb4358af3a4
I20260925 09:14:07.622936 8732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414: Bootstrap starting.
I20260925 09:14:07.626106 8732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:07.627178 8732 log.cc:824] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:07.630280 8732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414: No bootstrap required, opened a new log
I20260925 09:14:07.630724 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:07.618677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.3.19.126:42801)
0925 09:14:07.619661 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:07.619672 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:07.619712 (+ 40us) server_negotiation.cc:407] Connection header received
0925 09:14:07.621667 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:07.621693 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:07.622177 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:07.622569 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:07.623147 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:07.624500 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:07.625896 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:07.626665 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:07.626863 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:07.628160 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:07.628176 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:07.628194 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:07.628861 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:07.628941 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:07.628969 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:07.629055 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:07.629358 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 09:14:07.629707 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":353,"threads_started":1}
I20260925 09:14:07.634618 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:14:07.638836 8732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375eb9af3f5146438d7e0fd192b61414" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42801 } }
I20260925 09:14:07.639271 8732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:07.639384 8732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375eb9af3f5146438d7e0fd192b61414, State: Initialized, Role: FOLLOWER
I20260925 09:14:07.639951 8732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375eb9af3f5146438d7e0fd192b61414" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42801 } }
I20260925 09:14:07.640192 8732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:14:07.640336 8732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:14:07.640511 8732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:07.642910 8732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375eb9af3f5146438d7e0fd192b61414" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42801 } }
I20260925 09:14:07.643438 8732 leader_election.cc:302] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 375eb9af3f5146438d7e0fd192b61414; no voters:
I20260925 09:14:07.644052 8732 leader_election.cc:288] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:14:07.644235 8737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:07.645560 8737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [term 1 LEADER]: Becoming Leader. State: Replica: 375eb9af3f5146438d7e0fd192b61414, State: Running, Role: LEADER
I20260925 09:14:07.646135 8737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375eb9af3f5146438d7e0fd192b61414" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42801 } }
I20260925 09:14:07.646549 8732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:07.648957 8739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 375eb9af3f5146438d7e0fd192b61414. Latest consensus state: current_term: 1 leader_uuid: "375eb9af3f5146438d7e0fd192b61414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375eb9af3f5146438d7e0fd192b61414" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42801 } } }
I20260925 09:14:07.649425 8739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:07.652137 8743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:14:07.651926 8738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "375eb9af3f5146438d7e0fd192b61414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375eb9af3f5146438d7e0fd192b61414" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42801 } } }
I20260925 09:14:07.652303 8738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:07.658927 8743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:14:07.669624 8743 catalog_manager.cc:1347] Generated new cluster ID: ccbb6691b8674a169e8849bab0ab874a
I20260925 09:14:07.669767 8743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:14:07.696309 8743 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:14:07.697795 8743 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:14:07.711540 8743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 375eb9af3f5146438d7e0fd192b61414: Generated new TSK 0
I20260925 09:14:07.712445 8743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:14:07.788461 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:07.727299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.3.19.126:42801)
0925 09:14:07.727516 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:07.727524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:07.728010 (+ 486us) server_negotiation.cc:407] Connection header received
0925 09:14:07.728217 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:07.728223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:07.728319 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:07.728498 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0925 09:14:07.728946 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:07.730153 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:07.731603 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:07.732294 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:07.732422 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:07.733024 (+ 602us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 09:14:07.787658 (+ 54634us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 09:14:07.788118 (+ 460us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54012: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 09:14:07.788609 8735 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54012: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 09:14:07.788682 8761 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:07.727146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:42801 (local address 127.0.0.1:54012)
0925 09:14:07.727906 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:07.727922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:07.728046 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:07.728509 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:07.728512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:07.728522 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0925 09:14:07.728802 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:07.728810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:07.730339 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:07.730345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:07.731455 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:07.731466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:07.732557 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:07.732563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:07.732709 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:07.732746 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 09:14:07.788044 (+ 55298us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 09:14:07.788304 (+ 260us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.19.126:42801: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":472,"threads_started":1}
I20260925 09:14:07.790555 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 8666
I20260925 09:14:07.791877 8727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:07.890213 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 8666
2026-09-25T09:14:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (780 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (420 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (422 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:14:08 dist-test-slave-l6qh krb5kdc[8776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:14:08 dist-test-slave-l6qh krb5kdc[8776](info): set up 2 sockets
Sep 25 09:14:08 dist-test-slave-l6qh krb5kdc[8776](info): commencing operation
krb5kdc: starting...
W20260925 09:14:09.839597 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:14:09 dist-test-slave-l6qh krb5kdc[8776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:14:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:14:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:14:10.077543 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:35999
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:35999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:10.412740 8789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:10.413050 8789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:10.413221 8789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:10.420646 8789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:10.420766 8789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:10.420861 8789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:10.420941 8789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:10.421023 8789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:14:10.431197 8789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:35999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:35999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:10.432581 8789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:10.433947 8789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:10.446013 8794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:10.446067 8797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:10.445948 8795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:10.447001 8789 server_base.cc:1034] running on GCE node
I20260925 09:14:10.447806 8789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:10.449373 8789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:10.450655 8789 hybrid_clock.cc:648] HybridClock initialized: now 1790327650450596 us; error 56 us; skew 500 ppm
I20260925 09:14:10.451058 8789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:10 dist-test-slave-l6qh krb5kdc[8776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:10.454291 8789 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:14:10.455834 8789 webserver.cc:460] Webserver started at http://127.3.19.126:35707/ using document root <none> and password file <none>
I20260925 09:14:10.456562 8789 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:10.456708 8789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:10.457031 8789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:10.460207 8789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "ad03c74a22994a6890b111d4d8430f9c"
format_stamp: "Formatted at 2026-09-25 09:14:10 on dist-test-slave-l6qh"
server_key: "2cf4e6ef827e52c7ca3ed4124afdd69d"
server_key_iv: "c36ac500eea7a39a288ffb52fb57fee2"
server_key_version: "encryptionkey@0"
I20260925 09:14:10.461004 8789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "ad03c74a22994a6890b111d4d8430f9c"
format_stamp: "Formatted at 2026-09-25 09:14:10 on dist-test-slave-l6qh"
server_key: "2cf4e6ef827e52c7ca3ed4124afdd69d"
server_key_iv: "c36ac500eea7a39a288ffb52fb57fee2"
server_key_version: "encryptionkey@0"
I20260925 09:14:10.466526 8789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 09:14:10.470434 8804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:10.471921 8789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 09:14:10.472122 8789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "ad03c74a22994a6890b111d4d8430f9c"
format_stamp: "Formatted at 2026-09-25 09:14:10 on dist-test-slave-l6qh"
server_key: "2cf4e6ef827e52c7ca3ed4124afdd69d"
server_key_iv: "c36ac500eea7a39a288ffb52fb57fee2"
server_key_version: "encryptionkey@0"
I20260925 09:14:10.472332 8789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:10.523896 8789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:10.528276 8789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:10.528585 8789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:10.548090 8789 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:35999
I20260925 09:14:10.548147 8855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:35999 every 8 connection(s)
I20260925 09:14:10.549589 8789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:14:10.550082 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 8789
I20260925 09:14:10.550303 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:14:10.551147 3149 external_mini_cluster.cc:1428] Setting key 06deccc5a85478ede014fe3860d7fcb7
I20260925 09:14:10.554212 8856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:10.565292 8856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c: Bootstrap starting.
Sep 25 09:14:10 dist-test-slave-l6qh krb5kdc[8776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:14:10.569331 8856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:10.570484 8856 log.cc:824] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:10.573658 8856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c: No bootstrap required, opened a new log
I20260925 09:14:10.574970 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:10.553301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.3.19.126:35999)
0925 09:14:10.554122 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:10.554135 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:10.554177 (+ 42us) server_negotiation.cc:407] Connection header received
0925 09:14:10.556042 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:10.556066 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:10.556459 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:10.556910 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:10.558401 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:10.559816 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:10.561268 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:10.562013 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:10.562250 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:10.566783 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:10.566792 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:10.566808 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:10.566853 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:10.570474 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:10.571484 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:10.571492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:10.571499 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:10.571677 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:10.572096 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:10.572101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:10.572104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:10.572630 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:10.572944 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:10.573437 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 09:14:10.573913 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":367,"threads_started":1}
I20260925 09:14:10.582196 8856 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad03c74a22994a6890b111d4d8430f9c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 35999 } }
I20260925 09:14:10.582684 8856 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:10.582798 8856 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad03c74a22994a6890b111d4d8430f9c, State: Initialized, Role: FOLLOWER
I20260925 09:14:10.583369 8856 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad03c74a22994a6890b111d4d8430f9c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 35999 } }
I20260925 09:14:10.583597 8856 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:14:10.583751 8856 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:14:10.583933 8856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:10.586469 8856 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad03c74a22994a6890b111d4d8430f9c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 35999 } }
I20260925 09:14:10.587119 8856 leader_election.cc:302] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad03c74a22994a6890b111d4d8430f9c; no voters:
I20260925 09:14:10.587954 8856 leader_election.cc:288] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:14:10.588128 8861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:10.590130 8861 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [term 1 LEADER]: Becoming Leader. State: Replica: ad03c74a22994a6890b111d4d8430f9c, State: Running, Role: LEADER
I20260925 09:14:10.590416 8856 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:10.590729 8861 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad03c74a22994a6890b111d4d8430f9c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 35999 } }
I20260925 09:14:10.593580 8862 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad03c74a22994a6890b111d4d8430f9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad03c74a22994a6890b111d4d8430f9c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 35999 } } }
I20260925 09:14:10.593961 8862 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:10.594451 8863 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad03c74a22994a6890b111d4d8430f9c. Latest consensus state: current_term: 1 leader_uuid: "ad03c74a22994a6890b111d4d8430f9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad03c74a22994a6890b111d4d8430f9c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 35999 } } }
I20260925 09:14:10.594784 8863 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:10.596606 8869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:14:10.603603 8869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:14:10.613288 8869 catalog_manager.cc:1347] Generated new cluster ID: 1140b4bfeec346a69ba0c9f835cd11d0
I20260925 09:14:10.613497 8869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:14:10.627938 8869 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:14:10.629779 8869 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:14:10.639412 8869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad03c74a22994a6890b111d4d8430f9c: Generated new TSK 0
I20260925 09:14:10.640518 8869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:14:10.705183 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:35999
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:11.048035 8883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:11.048346 8883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:11.048491 8883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:11.055847 8883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:11.055974 8883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:11.056162 8883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:14:11.066205 8883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:35999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:11.067700 8883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:11.069067 8883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:11.082803 8889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:11.082870 8891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:11.082845 8888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:11.087033 8883 server_base.cc:1034] running on GCE node
I20260925 09:14:11.087791 8883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:11.088856 8883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:11.090101 8883 hybrid_clock.cc:648] HybridClock initialized: now 1790327651090045 us; error 50 us; skew 500 ppm
I20260925 09:14:11.090466 8883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:11 dist-test-slave-l6qh krb5kdc[8776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:11.094180 8883 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:14:11.095886 8883 webserver.cc:460] Webserver started at http://127.3.19.65:35657/ using document root <none> and password file <none>
I20260925 09:14:11.096777 8883 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:11.096927 8883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:11.097308 8883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:11.100577 8883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "71a998a51a0b4cc48eeca2c6e8fb08da"
format_stamp: "Formatted at 2026-09-25 09:14:11 on dist-test-slave-l6qh"
server_key: "864e3180cc292ebcc7e5d031742dfe65"
server_key_iv: "aa9351b71d0b84789f5ff7c7a17d37b5"
server_key_version: "encryptionkey@0"
I20260925 09:14:11.101514 8883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "71a998a51a0b4cc48eeca2c6e8fb08da"
format_stamp: "Formatted at 2026-09-25 09:14:11 on dist-test-slave-l6qh"
server_key: "864e3180cc292ebcc7e5d031742dfe65"
server_key_iv: "aa9351b71d0b84789f5ff7c7a17d37b5"
server_key_version: "encryptionkey@0"
I20260925 09:14:11.107105 8883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:14:11.111016 8898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:11.112566 8883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:14:11.112775 8883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "71a998a51a0b4cc48eeca2c6e8fb08da"
format_stamp: "Formatted at 2026-09-25 09:14:11 on dist-test-slave-l6qh"
server_key: "864e3180cc292ebcc7e5d031742dfe65"
server_key_iv: "aa9351b71d0b84789f5ff7c7a17d37b5"
server_key_version: "encryptionkey@0"
I20260925 09:14:11.113010 8883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:11.140889 8883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:11.145550 8883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:11.145821 8883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:11.146754 8883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:11.148057 8883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:11.148134 8883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:11.148218 8883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:11.148264 8883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:11.181341 8883 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:44539
I20260925 09:14:11.181342 9010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:44539 every 8 connection(s)
I20260925 09:14:11.182693 8883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:14:11.189790 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 8883
I20260925 09:14:11.190022 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:14:11.190698 3149 external_mini_cluster.cc:1428] Setting key ac641baae6030496edcffa1b5e07d44f
Sep 25 09:14:11 dist-test-slave-l6qh krb5kdc[8776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:14:11.205803 9013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:11.185283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:45485 (local address 127.3.19.126:35999)
0925 09:14:11.185744 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:11.185748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:11.186786 (+ 1038us) server_negotiation.cc:407] Connection header received
0925 09:14:11.188636 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:11.188642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:11.188737 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:11.188857 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:11.190628 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:11.191509 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:11.193687 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:11.194475 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:11.194652 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:11.198643 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:11.198648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:11.198651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:11.198685 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:11.202271 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:11.203130 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:11.203138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:11.203145 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:11.203296 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:11.203748 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:11.203755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:11.203761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:11.204150 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:11.204299 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:11.205033 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 09:14:11.205320 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":254,"threads_started":1}
I20260925 09:14:11.206574 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:11.185905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:35999 (local address 127.3.19.65:45485)
0925 09:14:11.186615 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:11.186659 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:11.188349 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:11.189108 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:11.189116 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:11.189728 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:11.190413 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:11.190431 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:11.191734 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:11.191738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:11.193539 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:11.193552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:11.194719 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:11.194724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:11.194927 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:11.195668 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:11.195692 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:11.198328 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:11.202466 (+ 4138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:11.202475 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:11.202491 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:11.202973 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:11.203408 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:11.203411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:11.203413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:11.203626 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:11.204267 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:11.204273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:11.204636 (+ 363us) client_negotiation.cc:769] Sending connection context
0925 09:14:11.204859 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 09:14:11.205354 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":285,"threads_started":1}
I20260925 09:14:11.208204 9011 heartbeater.cc:347] Connected to a master server at 127.3.19.126:35999
I20260925 09:14:11.208530 9011 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:11.209434 9011 heartbeater.cc:511] Master 127.3.19.126:35999 requested a full tablet report, sending...
I20260925 09:14:11.211570 8821 ts_manager.cc:194] Registered new tserver with Master: 71a998a51a0b4cc48eeca2c6e8fb08da (127.3.19.65:44539)
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
I20260925 09:14:11.213106 8821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:45485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:14:11.240453 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:35999
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:11.572866 9017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:11.573230 9017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:11.573424 9017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:11.581107 9017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:11.581328 9017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:11.581519 9017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:14:11.592517 9017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:35999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:11.594151 9017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:11.595444 9017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:11.609885 9025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:11.609853 9022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:11.609925 9023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:11.611183 9017 server_base.cc:1034] running on GCE node
I20260925 09:14:11.611830 9017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:11.612828 9017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:11.614045 9017 hybrid_clock.cc:648] HybridClock initialized: now 1790327651613996 us; error 42 us; skew 500 ppm
I20260925 09:14:11.614399 9017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:11 dist-test-slave-l6qh krb5kdc[8776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:11.617931 9017 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:14:11.619712 9017 webserver.cc:460] Webserver started at http://127.3.19.66:33577/ using document root <none> and password file <none>
I20260925 09:14:11.620496 9017 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:11.620690 9017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:11.621058 9017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:11.624306 9017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "2941a7893b814384b7225265701cd338"
format_stamp: "Formatted at 2026-09-25 09:14:11 on dist-test-slave-l6qh"
server_key: "c83d69e7f977000cc4c9971cbaf75d33"
server_key_iv: "601d6b1b99e38f732e31d3c7ba7a6bb8"
server_key_version: "encryptionkey@0"
I20260925 09:14:11.625165 9017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "2941a7893b814384b7225265701cd338"
format_stamp: "Formatted at 2026-09-25 09:14:11 on dist-test-slave-l6qh"
server_key: "c83d69e7f977000cc4c9971cbaf75d33"
server_key_iv: "601d6b1b99e38f732e31d3c7ba7a6bb8"
server_key_version: "encryptionkey@0"
I20260925 09:14:11.631536 9017 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260925 09:14:11.635422 9032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:11.637012 9017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:14:11.637265 9017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "2941a7893b814384b7225265701cd338"
format_stamp: "Formatted at 2026-09-25 09:14:11 on dist-test-slave-l6qh"
server_key: "c83d69e7f977000cc4c9971cbaf75d33"
server_key_iv: "601d6b1b99e38f732e31d3c7ba7a6bb8"
server_key_version: "encryptionkey@0"
I20260925 09:14:11.637526 9017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:11.653563 9017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:11.659206 9017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:11.659536 9017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:11.660696 9017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:11.662346 9017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:11.662428 9017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:11.662576 9017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:11.662639 9017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:11.695817 9017 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:32993
I20260925 09:14:11.695911 9144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:32993 every 8 connection(s)
I20260925 09:14:11.697142 9017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:14:11.702246 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 9017
I20260925 09:14:11.702447 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:14:11.702863 3149 external_mini_cluster.cc:1428] Setting key e21743cdd35d2a26eee3bd3690dd7719
Sep 25 09:14:11 dist-test-slave-l6qh krb5kdc[8776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:14:11.718417 9013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:11.699951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:35933 (local address 127.3.19.126:35999)
0925 09:14:11.700126 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:11.700132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:11.701401 (+ 1269us) server_negotiation.cc:407] Connection header received
0925 09:14:11.703403 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:11.703409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:11.703517 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:11.703642 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:11.705181 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:11.705943 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:11.707544 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:11.707988 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:11.708120 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:11.712315 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:11.712320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:11.712323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:11.712356 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:11.715119 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:11.715871 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:11.715881 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:11.715884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:11.716008 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:11.716472 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:11.716477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:11.716480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:11.716824 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:11.716987 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:11.717828 (+ 841us) server_negotiation.cc:299] Negotiation successful
0925 09:14:11.718068 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 09:14:11.719547 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:11.700466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:35999 (local address 127.3.19.66:35933)
0925 09:14:11.701193 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:11.701282 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:11.703127 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:11.703884 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:11.703893 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:11.704413 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:11.705010 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:11.705027 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:11.706110 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:11.706115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:11.707383 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:11.707395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:11.708228 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:11.708235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:11.708477 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:11.709209 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:11.709282 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:11.712056 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:11.715277 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:11.715286 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:11.715305 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:11.715735 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:11.716120 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:11.716125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:11.716130 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:11.716354 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:11.716955 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:11.716963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:11.717433 (+ 470us) client_negotiation.cc:769] Sending connection context
0925 09:14:11.717740 (+ 307us) client_negotiation.cc:240] Negotiation successful
0925 09:14:11.718303 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":292,"threads_started":1}
I20260925 09:14:11.721212 9145 heartbeater.cc:347] Connected to a master server at 127.3.19.126:35999
I20260925 09:14:11.721622 9145 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:11.722657 9145 heartbeater.cc:511] Master 127.3.19.126:35999 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
I20260925 09:14:11.724452 8821 ts_manager.cc:194] Registered new tserver with Master: 2941a7893b814384b7225265701cd338 (127.3.19.66:32993)
I20260925 09:14:11.725431 8821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:35933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:14:11.753571 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:35999
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:12.086575 9150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:12.086871 9150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:12.086966 9150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:12.094413 9150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:12.094537 9150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:12.094650 9150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:14:12.104790 9150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:35999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:12.106114 9150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:12.107376 9150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:12.121377 9155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:12.121505 9156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:12.121693 9158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:12.126865 9150 server_base.cc:1034] running on GCE node
I20260925 09:14:12.127524 9150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:12.128505 9150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:12.129705 9150 hybrid_clock.cc:648] HybridClock initialized: now 1790327652129624 us; error 73 us; skew 500 ppm
I20260925 09:14:12.130052 9150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:12.133289 9150 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:14:12.135047 9150 webserver.cc:460] Webserver started at http://127.3.19.67:41253/ using document root <none> and password file <none>
I20260925 09:14:12.135800 9150 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:12.135944 9150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:12.136261 9150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:12.139312 9150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "ed4ed424b1944980af33e1934d44e86c"
format_stamp: "Formatted at 2026-09-25 09:14:12 on dist-test-slave-l6qh"
server_key: "7fc966fdd22e9a6c1ec645beefe786c3"
server_key_iv: "c249b7d4afef459348ca62d87690293d"
server_key_version: "encryptionkey@0"
I20260925 09:14:12.140142 9150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "ed4ed424b1944980af33e1934d44e86c"
format_stamp: "Formatted at 2026-09-25 09:14:12 on dist-test-slave-l6qh"
server_key: "7fc966fdd22e9a6c1ec645beefe786c3"
server_key_iv: "c249b7d4afef459348ca62d87690293d"
server_key_version: "encryptionkey@0"
I20260925 09:14:12.145746 9150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 09:14:12.149369 9165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:12.150782 9150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:14:12.150969 9150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "ed4ed424b1944980af33e1934d44e86c"
format_stamp: "Formatted at 2026-09-25 09:14:12 on dist-test-slave-l6qh"
server_key: "7fc966fdd22e9a6c1ec645beefe786c3"
server_key_iv: "c249b7d4afef459348ca62d87690293d"
server_key_version: "encryptionkey@0"
I20260925 09:14:12.151192 9150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:12.181000 9150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:12.185395 9150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:12.185694 9150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:12.186654 9150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:12.187960 9150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:12.188063 9150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:12.188206 9150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:12.188283 9150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:12.216706 9011 heartbeater.cc:503] Master 127.3.19.126:35999 was elected leader, sending a full tablet report...
I20260925 09:14:12.224630 9150 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:46811
I20260925 09:14:12.224675 9277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:46811 every 8 connection(s)
I20260925 09:14:12.225957 9150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:14:12.235746 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 9150
I20260925 09:14:12.235924 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:14:12.236493 3149 external_mini_cluster.cc:1428] Setting key 55e34cd7f804b04634ec6f94c5cdace9
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:14:12.246719 9280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:12.228528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:46823 (local address 127.3.19.126:35999)
0925 09:14:12.229000 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:12.229004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:12.230000 (+ 996us) server_negotiation.cc:407] Connection header received
0925 09:14:12.231823 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:12.231828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:12.231922 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:12.232046 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:12.233672 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.234493 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.236625 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.237106 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.237280 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:12.241143 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:12.241148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:12.241151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:12.241185 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:12.243949 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.244703 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.244707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.244709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.244825 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.245202 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.245205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.245207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.245516 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:12.245624 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:12.246292 (+ 668us) server_negotiation.cc:299] Negotiation successful
0925 09:14:12.246451 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":257,"threads_started":1}
I20260925 09:14:12.247721 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:12.229028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:35999 (local address 127.3.19.67:46823)
0925 09:14:12.229825 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:12.229865 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:12.231538 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:12.232301 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:12.232310 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:12.232842 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:12.233496 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:12.233515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.234665 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.234669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:12.236417 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:12.236430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.237398 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.237406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:12.237695 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:12.238436 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:12.238459 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:12.240881 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:12.244103 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:12.244111 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:12.244125 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:12.244593 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:12.244916 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:12.244919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:12.244921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:12.245098 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:12.245627 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:12.245633 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:12.245979 (+ 346us) client_negotiation.cc:769] Sending connection context
0925 09:14:12.246202 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 09:14:12.246634 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":397,"threads_started":1}
I20260925 09:14:12.249167 9278 heartbeater.cc:347] Connected to a master server at 127.3.19.126:35999
I20260925 09:14:12.249511 9278 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:12.250276 9278 heartbeater.cc:511] Master 127.3.19.126:35999 requested a full tablet report, sending...
I20260925 09:14:12.252151 8820 ts_manager.cc:194] Registered new tserver with Master: ed4ed424b1944980af33e1934d44e86c (127.3.19.67:46811)
I20260925 09:14:12.253139 8820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:46823
I20260925 09:14:12.264868 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/test-admin.keytab.
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:12.292670 3149 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 09:14:12.298162 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:12.295465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.3.19.65:44539)
0925 09:14:12.296209 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:12.296217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:12.296243 (+ 26us) server_negotiation.cc:407] Connection header received
0925 09:14:12.296304 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:12.296310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:12.296533 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:12.296671 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:12.297725 (+ 1054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37324: BlockingRecv error: recv got EOF from 127.0.0.1:37324 (error 108)
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":420,"threads_started":1}
W20260925 09:14:12.298198 3149 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/krb5cc' of type FILE
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:12.299844 3149 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:12.301172 3149 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:14:12.316990 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:12.302973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.3.19.65:44539)
0925 09:14:12.303173 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:12.303180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:12.303334 (+ 154us) server_negotiation.cc:407] Connection header received
0925 09:14:12.303577 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:12.303583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:12.303654 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:12.303758 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:12.304752 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.305781 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.307005 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.307580 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.307724 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:12.311333 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:12.311344 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:12.311358 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:12.311394 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:12.314300 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.314963 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.314970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.314973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.315088 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.315458 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.315463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.315466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.315816 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:12.316207 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:12.316426 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 09:14:12.316687 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 09:14:12.323148 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:12.321286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.3.19.65:44539)
0925 09:14:12.321513 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:12.321521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:12.321794 (+ 273us) server_negotiation.cc:407] Connection header received
0925 09:14:12.321990 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:12.321995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:12.322067 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:12.322167 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:12.322901 (+ 734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37346: BlockingRecv error: recv got EOF from 127.0.0.1:37346 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 09:14:12.323339 3149 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/krb5cc' of type FILE
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:12.324759 3149 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:12.326061 3149 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:14:12.340456 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:12.327313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.3.19.65:44539)
0925 09:14:12.327515 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:12.327521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:12.327695 (+ 174us) server_negotiation.cc:407] Connection header received
0925 09:14:12.327809 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:12.327812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:12.327867 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:12.327959 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:12.328804 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.329579 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.331030 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.331457 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.331553 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:12.335196 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:12.335206 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:12.335214 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:12.335268 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:12.338154 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.338776 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.338780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.338782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.338925 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.339271 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.339274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.339276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.339543 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:12.339734 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:12.339960 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 09:14:12.340114 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260925 09:14:12.345063 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:12.343399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.3.19.65:44539)
0925 09:14:12.343615 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:12.343620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:12.343845 (+ 225us) server_negotiation.cc:407] Connection header received
0925 09:14:12.344002 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:12.344006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:12.344077 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:12.344171 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:12.344827 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37354: BlockingRecv error: recv got EOF from 127.0.0.1:37354 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 09:14:12.345175 3149 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790327600440376-3149-0/krb5kdc/krb5cc' of type FILE
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:12.346784 3149 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:12.348044 3149 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:14:12.362553 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:12.349328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.3.19.65:44539)
0925 09:14:12.349542 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:12.349549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:12.349829 (+ 280us) server_negotiation.cc:407] Connection header received
0925 09:14:12.350011 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:12.350016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:12.350089 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:12.350191 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:12.351143 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.351856 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.353096 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.353611 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.353760 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:12.357343 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:12.357349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:12.357354 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:12.357398 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:12.359850 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.360533 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.360541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.360545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.360687 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.361059 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.361065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.361069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.361431 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:12.361744 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:12.362011 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 09:14:12.362223 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 09:14:12.364156 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 8883
I20260925 09:14:12.376502 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:12.364883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.3.19.65:44539)
0925 09:14:12.365069 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:12.365073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:12.365444 (+ 371us) server_negotiation.cc:407] Connection header received
0925 09:14:12.365584 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:12.365588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:12.365643 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:12.365726 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:12.366703 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.367363 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.368544 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.368933 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.369037 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:12.371268 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:12.371273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:12.371276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:12.371306 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:12.373886 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.374572 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.374576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.374579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.374706 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.375124 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.375130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.375133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.375386 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:12.375578 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:12.376003 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 09:14:12.376214 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 09:14:12.377157 9006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:12.460409 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 8883
I20260925 09:14:12.477150 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 9017
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:14:12.492563 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:12.478062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.3.19.66:32993)
0925 09:14:12.478586 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:12.478593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:12.478619 (+ 26us) server_negotiation.cc:407] Connection header received
0925 09:14:12.478679 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:12.478686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:12.478896 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:12.479033 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:12.479934 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.480915 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.482235 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.482803 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.482912 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:12.486648 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:12.486660 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:12.486676 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:12.486713 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:12.489778 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.490525 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.490532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.490534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.490673 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.491144 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.491149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.491151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.491537 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:12.491764 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:12.492070 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 09:14:12.492271 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":267,"threads_started":1}
I20260925 09:14:12.494024 9140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:12.575758 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 9017
I20260925 09:14:12.591992 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 9150
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[8776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:14:12.609377 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:12.593091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.3.19.67:46811)
0925 09:14:12.593946 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:12.593970 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:12.594011 (+ 41us) server_negotiation.cc:407] Connection header received
0925 09:14:12.594091 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:12.594100 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:12.594395 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:12.594602 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:12.595946 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.597486 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.598813 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:12.599458 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:12.599576 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:12.603371 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:12.603384 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:12.603399 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:12.603436 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:12.606479 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.607198 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.607205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.607207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.607338 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:12.607833 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:12.607837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:12.607839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:12.608216 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:12.608450 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:12.608798 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 09:14:12.609011 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":451,"threads_started":1}
I20260925 09:14:12.610831 9273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:12.693674 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 9150
I20260925 09:14:12.710222 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 8789
I20260925 09:14:12.711410 8851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:12.814999 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 8789
2026-09-25T09:14:12Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4087 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[9306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[9306](info): set up 2 sockets
Sep 25 09:14:12 dist-test-slave-l6qh krb5kdc[9306](info): commencing operation
krb5kdc: starting...
W20260925 09:14:13.927006 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:14:14 dist-test-slave-l6qh krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:14:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:14:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.126.keytab.
Entry for principal oryx/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.126.keytab.
I20260925 09:14:14.161345 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:39883
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.126.keytab
--principal=oryx/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:39883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:14.491006 9319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:14.491319 9319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:14.491410 9319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:14.498805 9319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:14.498932 9319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:14.499043 9319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:14.499207 9319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:14.499266 9319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:14:14.509265 9319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:39883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.126.keytab
--principal=oryx/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:39883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:14.510697 9319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:14.512079 9319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:14.524760 9325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:14.524731 9327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:14.525425 9324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:14.527331 9319 server_base.cc:1034] running on GCE node
I20260925 09:14:14.528439 9319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:14.530260 9319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:14.531575 9319 hybrid_clock.cc:648] HybridClock initialized: now 1790327654531514 us; error 47 us; skew 500 ppm
I20260925 09:14:14.532037 9319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:14 dist-test-slave-l6qh krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327654, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:14.535831 9319 init.cc:377] Logged in from keytab as oryx/127.3.19.126@KRBTEST.COM (short username oryx)
I20260925 09:14:14.537545 9319 webserver.cc:460] Webserver started at http://127.3.19.126:37783/ using document root <none> and password file <none>
I20260925 09:14:14.538321 9319 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:14.538477 9319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:14.538798 9319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:14.542008 9319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "3cd59f740532408ba0b9dd8b2012096a"
format_stamp: "Formatted at 2026-09-25 09:14:14 on dist-test-slave-l6qh"
server_key: "2f859561475d324a83cc25028ced1288"
server_key_iv: "c86b2aefd365cbfaae33072ea1ec1bd2"
server_key_version: "encryptionkey@0"
I20260925 09:14:14.542820 9319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "3cd59f740532408ba0b9dd8b2012096a"
format_stamp: "Formatted at 2026-09-25 09:14:14 on dist-test-slave-l6qh"
server_key: "2f859561475d324a83cc25028ced1288"
server_key_iv: "c86b2aefd365cbfaae33072ea1ec1bd2"
server_key_version: "encryptionkey@0"
I20260925 09:14:14.548288 9319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 09:14:14.552225 9334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:14.553968 9319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:14.554206 9319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "3cd59f740532408ba0b9dd8b2012096a"
format_stamp: "Formatted at 2026-09-25 09:14:14 on dist-test-slave-l6qh"
server_key: "2f859561475d324a83cc25028ced1288"
server_key_iv: "c86b2aefd365cbfaae33072ea1ec1bd2"
server_key_version: "encryptionkey@0"
I20260925 09:14:14.554450 9319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:14.575129 9319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:14.579962 9319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:14.580287 9319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:14.599738 9319 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:39883
I20260925 09:14:14.599790 9385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:39883 every 8 connection(s)
I20260925 09:14:14.601161 9319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:14:14.604156 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 9319
I20260925 09:14:14.604374 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:14:14.605203 3149 external_mini_cluster.cc:1428] Setting key 05afbf4b6d771860a9e60f28a6c738a2
I20260925 09:14:14.606258 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 09:14:14 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:14.622067 9386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a: Bootstrap starting.
I20260925 09:14:14.626933 9386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:14.628298 9386 log.cc:824] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:14.632161 9386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a: No bootstrap required, opened a new log
I20260925 09:14:14.632184 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:14.607349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.3.19.126:39883)
0925 09:14:14.608151 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:14.608172 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:14.608213 (+ 41us) server_negotiation.cc:407] Connection header received
0925 09:14:14.609846 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:14.609870 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:14.610279 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:14.610630 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:14.612687 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:14.614238 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:14.616160 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:14.617270 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:14.617507 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:14.622342 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:14.622355 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:14.622377 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:14.622453 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:14.626900 (+ 4447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:14.627776 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:14.627786 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:14.627796 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:14.628036 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:14.628432 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:14.628437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:14.628441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:14.629215 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:14.629578 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:14.630136 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 09:14:14.630677 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":454,"threads_started":1}
I20260925 09:14:14.641455 9386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } }
I20260925 09:14:14.641856 9386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:14.641959 9386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd59f740532408ba0b9dd8b2012096a, State: Initialized, Role: FOLLOWER
I20260925 09:14:14.642596 9386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } }
I20260925 09:14:14.642820 9386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:14:14.642933 9386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:14:14.643075 9386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:14.645501 9386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } }
I20260925 09:14:14.646128 9386 leader_election.cc:302] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cd59f740532408ba0b9dd8b2012096a; no voters:
I20260925 09:14:14.646843 9386 leader_election.cc:288] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:14:14.646956 9391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:14.647879 9391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [term 1 LEADER]: Becoming Leader. State: Replica: 3cd59f740532408ba0b9dd8b2012096a, State: Running, Role: LEADER
I20260925 09:14:14.648576 9391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } }
I20260925 09:14:14.649418 9386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:14.651356 9393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cd59f740532408ba0b9dd8b2012096a. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } }
I20260925 09:14:14.651849 9393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:14.652199 9392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } }
I20260925 09:14:14.652676 9392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:14.654932 9396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:14:14.660755 9396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:14:14.672072 9396 catalog_manager.cc:1347] Generated new cluster ID: c0376b7b833547c7924a8ec1c3637e51
I20260925 09:14:14.672204 9396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:14:14.695701 9396 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:14:14.697077 9396 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:14:14.706315 9396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a: Generated new TSK 0
I20260925 09:14:14.707281 9396 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab.
Entry for principal oryx/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab.
I20260925 09:14:14.777037 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:39883
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab
--principal=oryx/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:15.135754 9412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:15.136070 9412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:15.136162 9412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:15.143658 9412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:15.143787 9412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:15.143900 9412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:14:15.154117 9412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab
--principal=oryx/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:39883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:15.155508 9412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:15.156922 9412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:15.173058 9418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:15.173135 9417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:15.173110 9420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:15.174015 9412 server_base.cc:1034] running on GCE node
I20260925 09:14:15.174723 9412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:15.175697 9412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:15.176909 9412 hybrid_clock.cc:648] HybridClock initialized: now 1790327655176832 us; error 62 us; skew 500 ppm
I20260925 09:14:15.177194 9412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:15 dist-test-slave-l6qh krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327655, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:15.184566 9412 init.cc:377] Logged in from keytab as oryx/127.3.19.65@KRBTEST.COM (short username oryx)
I20260925 09:14:15.186295 9412 webserver.cc:460] Webserver started at http://127.3.19.65:44921/ using document root <none> and password file <none>
I20260925 09:14:15.187016 9412 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:15.187120 9412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:15.187369 9412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:15.190481 9412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "ff75d2f51e324151a2b0a4e0705a5fcb"
format_stamp: "Formatted at 2026-09-25 09:14:15 on dist-test-slave-l6qh"
server_key: "11e94c0634f5eabe08a438af0037dd18"
server_key_iv: "06ccf07e76936fba8a5332250262aa8b"
server_key_version: "encryptionkey@0"
I20260925 09:14:15.191234 9412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "ff75d2f51e324151a2b0a4e0705a5fcb"
format_stamp: "Formatted at 2026-09-25 09:14:15 on dist-test-slave-l6qh"
server_key: "11e94c0634f5eabe08a438af0037dd18"
server_key_iv: "06ccf07e76936fba8a5332250262aa8b"
server_key_version: "encryptionkey@0"
I20260925 09:14:15.196630 9412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:14:15.200398 9427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:15.202091 9412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:15.202243 9412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "ff75d2f51e324151a2b0a4e0705a5fcb"
format_stamp: "Formatted at 2026-09-25 09:14:15 on dist-test-slave-l6qh"
server_key: "11e94c0634f5eabe08a438af0037dd18"
server_key_iv: "06ccf07e76936fba8a5332250262aa8b"
server_key_version: "encryptionkey@0"
I20260925 09:14:15.202438 9412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:15.223192 9412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:15.227877 9412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:15.228133 9412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:15.229055 9412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:15.230515 9412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:15.230592 9412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:15.230680 9412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:15.230724 9412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:15.265257 9412 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:39165
I20260925 09:14:15.265375 9539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:39165 every 8 connection(s)
I20260925 09:14:15.266619 9412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:14:15.275002 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 9412
I20260925 09:14:15.275238 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:14:15.276115 3149 external_mini_cluster.cc:1428] Setting key 3bc3662c1edfc094228e12852a1df732
Sep 25 09:14:15 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327655, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:15.288940 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:15.269307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:56743 (local address 127.3.19.126:39883)
0925 09:14:15.269769 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:15.269776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:15.270811 (+ 1035us) server_negotiation.cc:407] Connection header received
0925 09:14:15.272637 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:15.272641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:15.272712 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:15.272815 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:15.274489 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:15.275257 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:15.277587 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:15.278265 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:15.278408 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:15.283013 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:15.283018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:15.283021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:15.283055 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:15.285768 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:15.286627 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:15.286639 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:15.286642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:15.286754 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:15.287194 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:15.287200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:15.287203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:15.287468 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:15.287622 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:15.288351 (+ 729us) server_negotiation.cc:299] Negotiation successful
0925 09:14:15.288572 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":222,"threads_started":1}
I20260925 09:14:15.290066 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:15.269920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.65:56743)
0925 09:14:15.270631 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:15.270669 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:15.272331 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:15.273075 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:15.273083 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:15.273718 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:15.274307 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:15.274325 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:15.275456 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:15.275464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:15.277438 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:15.277450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:15.278561 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:15.278570 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:15.278902 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:15.279743 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:15.279792 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:15.282735 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:15.285990 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:15.286000 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:15.286020 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:15.286483 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:15.286871 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:15.286876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:15.286879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:15.287071 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:15.287599 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:15.287607 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:15.287969 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 09:14:15.288253 (+ 284us) client_negotiation.cc:240] Negotiation successful
0925 09:14:15.288820 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":274,"threads_started":1}
I20260925 09:14:15.291764 9540 heartbeater.cc:347] Connected to a master server at 127.3.19.126:39883
I20260925 09:14:15.292128 9540 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:15.293136 9540 heartbeater.cc:511] Master 127.3.19.126:39883 requested a full tablet report, sending...
I20260925 09:14:15.295801 9351 ts_manager.cc:194] Registered new tserver with Master: ff75d2f51e324151a2b0a4e0705a5fcb (127.3.19.65:39165)
I20260925 09:14:15.297537 9351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:56743
WARNING: no policy specified for oryx/127.3.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab.
Entry for principal oryx/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab.
I20260925 09:14:15.329938 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:39883
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab
--principal=oryx/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:15.672307 9546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:15.672626 9546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:15.672788 9546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:15.680735 9546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:15.680883 9546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:15.681167 9546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:14:15.692193 9546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab
--principal=oryx/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:39883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:15.693743 9546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:15.695219 9546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:15.709358 9554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:15.709619 9551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:15.709430 9552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:15.710704 9546 server_base.cc:1034] running on GCE node
I20260925 09:14:15.711464 9546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:15.712532 9546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:15.713790 9546 hybrid_clock.cc:648] HybridClock initialized: now 1790327655713736 us; error 66 us; skew 500 ppm
I20260925 09:14:15.714162 9546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:15 dist-test-slave-l6qh krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327655, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:15.717609 9546 init.cc:377] Logged in from keytab as oryx/127.3.19.66@KRBTEST.COM (short username oryx)
I20260925 09:14:15.719077 9546 webserver.cc:460] Webserver started at http://127.3.19.66:40533/ using document root <none> and password file <none>
I20260925 09:14:15.719811 9546 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:15.719914 9546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:15.720199 9546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:15.723476 9546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "80c500977f6c46049c7d88b40d4c582d"
format_stamp: "Formatted at 2026-09-25 09:14:15 on dist-test-slave-l6qh"
server_key: "3b84ebc37cf28021cdccd0f79eb28851"
server_key_iv: "7d1f31dc7b238b1d544d791547811870"
server_key_version: "encryptionkey@0"
I20260925 09:14:15.724242 9546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "80c500977f6c46049c7d88b40d4c582d"
format_stamp: "Formatted at 2026-09-25 09:14:15 on dist-test-slave-l6qh"
server_key: "3b84ebc37cf28021cdccd0f79eb28851"
server_key_iv: "7d1f31dc7b238b1d544d791547811870"
server_key_version: "encryptionkey@0"
I20260925 09:14:15.730134 9546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 09:14:15.734038 9561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:15.735545 9546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:14:15.735737 9546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "80c500977f6c46049c7d88b40d4c582d"
format_stamp: "Formatted at 2026-09-25 09:14:15 on dist-test-slave-l6qh"
server_key: "3b84ebc37cf28021cdccd0f79eb28851"
server_key_iv: "7d1f31dc7b238b1d544d791547811870"
server_key_version: "encryptionkey@0"
I20260925 09:14:15.735985 9546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:15.757206 9546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:15.762037 9546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:15.762364 9546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:15.763437 9546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:15.764855 9546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:15.764931 9546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:15.765074 9546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:15.765125 9546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:15.799170 9546 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:36105
I20260925 09:14:15.799206 9673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:36105 every 8 connection(s)
I20260925 09:14:15.800518 9546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:14:15.802059 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 9546
I20260925 09:14:15.802225 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:14:15.802747 3149 external_mini_cluster.cc:1428] Setting key 11aec1e956d8aa0be7e6faddb498a27b
Sep 25 09:14:15 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327655, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:15.825062 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:15.803420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:57255 (local address 127.3.19.126:39883)
0925 09:14:15.804013 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:15.804019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:15.805055 (+ 1036us) server_negotiation.cc:407] Connection header received
0925 09:14:15.807042 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:15.807048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:15.807132 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:15.807276 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:15.808945 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:15.809787 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:15.811954 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:15.812429 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:15.812553 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:15.817267 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:15.817276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:15.817284 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:15.817335 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:15.821354 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:15.822310 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:15.822319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:15.822325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:15.822502 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:15.823036 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:15.823043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:15.823049 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:15.823473 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:15.823656 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:15.824362 (+ 706us) server_negotiation.cc:299] Negotiation successful
0925 09:14:15.824615 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":351,"threads_started":1}
I20260925 09:14:15.825913 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:15.804049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.66:57255)
0925 09:14:15.804895 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:15.804948 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:15.806763 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:15.807573 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:15.807587 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:15.808155 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:15.808751 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:15.808769 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:15.809965 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:15.809971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:15.811727 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:15.811750 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:15.812689 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:15.812699 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:15.813066 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:15.813959 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:15.814017 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:15.816928 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:15.821570 (+ 4642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:15.821579 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:15.821595 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:15.822140 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:15.822690 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:15.822694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:15.822696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:15.822905 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:15.823608 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:15.823615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:15.823964 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 09:14:15.824234 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 09:14:15.824660 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":318,"threads_started":1}
WARNING: no policy specified for oryx/127.3.19.67@KRBTEST.COM; defaulting to no policy
I20260925 09:14:15.827714 9674 heartbeater.cc:347] Connected to a master server at 127.3.19.126:39883
I20260925 09:14:15.828042 9674 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:15.828884 9674 heartbeater.cc:511] Master 127.3.19.126:39883 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.19.67@KRBTEST.COM" created.
I20260925 09:14:15.830667 9351 ts_manager.cc:194] Registered new tserver with Master: 80c500977f6c46049c7d88b40d4c582d (127.3.19.66:36105)
I20260925 09:14:15.831957 9351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:57255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab.
Entry for principal oryx/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab.
I20260925 09:14:15.858224 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:39883
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab
--principal=oryx/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:16.190548 9680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:16.190843 9680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:16.191013 9680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:16.198365 9680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:16.198521 9680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:16.198712 9680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:14:16.208814 9680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab
--principal=oryx/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:39883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:16.210206 9680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:16.211550 9680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:16.224823 9685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:16.224895 9686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:16.224848 9688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:16.225731 9680 server_base.cc:1034] running on GCE node
I20260925 09:14:16.226408 9680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:16.227489 9680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:16.228785 9680 hybrid_clock.cc:648] HybridClock initialized: now 1790327656228719 us; error 62 us; skew 500 ppm
I20260925 09:14:16.229090 9680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:16 dist-test-slave-l6qh krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327656, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:16.232211 9680 init.cc:377] Logged in from keytab as oryx/127.3.19.67@KRBTEST.COM (short username oryx)
I20260925 09:14:16.233805 9680 webserver.cc:460] Webserver started at http://127.3.19.67:44927/ using document root <none> and password file <none>
I20260925 09:14:16.234522 9680 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:16.234651 9680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:16.234958 9680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:16.238276 9680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "594d6f796cae43afb1ca62740a564f50"
format_stamp: "Formatted at 2026-09-25 09:14:16 on dist-test-slave-l6qh"
server_key: "3afcfa3e67448a96657af7d397ace61e"
server_key_iv: "fbb96eb4bb0db88a38ddb3b3e5e584da"
server_key_version: "encryptionkey@0"
I20260925 09:14:16.239102 9680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "594d6f796cae43afb1ca62740a564f50"
format_stamp: "Formatted at 2026-09-25 09:14:16 on dist-test-slave-l6qh"
server_key: "3afcfa3e67448a96657af7d397ace61e"
server_key_iv: "fbb96eb4bb0db88a38ddb3b3e5e584da"
server_key_version: "encryptionkey@0"
I20260925 09:14:16.244659 9680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:14:16.248186 9695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:16.249871 9680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:16.250087 9680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "594d6f796cae43afb1ca62740a564f50"
format_stamp: "Formatted at 2026-09-25 09:14:16 on dist-test-slave-l6qh"
server_key: "3afcfa3e67448a96657af7d397ace61e"
server_key_iv: "fbb96eb4bb0db88a38ddb3b3e5e584da"
server_key_version: "encryptionkey@0"
I20260925 09:14:16.250310 9680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:16.276894 9680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:16.281780 9680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:16.282111 9680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:16.283200 9680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:16.284646 9680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:16.284720 9680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:16.284849 9680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:16.284909 9680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:16.301018 9540 heartbeater.cc:503] Master 127.3.19.126:39883 was elected leader, sending a full tablet report...
I20260925 09:14:16.319867 9680 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:35667
I20260925 09:14:16.319890 9807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:35667 every 8 connection(s)
I20260925 09:14:16.321185 9680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:14:16.331231 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 9680
I20260925 09:14:16.331410 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:14:16.331867 3149 external_mini_cluster.cc:1428] Setting key 10d6d0144d6ea0bc4f50ddf9bd86cc34
Sep 25 09:14:16 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327656, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:16.342453 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:16.323836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:60793 (local address 127.3.19.126:39883)
0925 09:14:16.324015 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:16.324019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:16.325135 (+ 1116us) server_negotiation.cc:407] Connection header received
0925 09:14:16.327180 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:16.327185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:16.327269 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:16.327392 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:16.329222 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.330029 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.331903 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.332419 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.332585 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:16.336854 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:16.336859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:16.336863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:16.336917 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:16.339501 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:16.340271 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:16.340282 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:16.340285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:16.340398 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:16.340834 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:16.340837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:16.340839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:16.341129 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:16.341348 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:16.342035 (+ 687us) server_negotiation.cc:299] Negotiation successful
0925 09:14:16.342212 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 09:14:16.343523 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:16.324250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.67:60793)
0925 09:14:16.324955 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:16.324998 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:16.326872 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:16.327800 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:16.327814 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:16.328413 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:16.329023 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:16.329042 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.330197 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.330201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:16.331714 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:16.331728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.332681 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.332690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:16.333022 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:16.333923 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:16.333947 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:16.336628 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:16.339679 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:16.339686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:16.339700 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:16.340132 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:16.340525 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:16.340532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:16.340535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:16.340728 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:16.341299 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:16.341305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:16.341680 (+ 375us) client_negotiation.cc:769] Sending connection context
0925 09:14:16.341918 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 09:14:16.342322 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":280,"threads_started":1}
I20260925 09:14:16.344995 9808 heartbeater.cc:347] Connected to a master server at 127.3.19.126:39883
I20260925 09:14:16.345377 9808 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:16.346151 9808 heartbeater.cc:511] Master 127.3.19.126:39883 requested a full tablet report, sending...
I20260925 09:14:16.347738 9351 ts_manager.cc:194] Registered new tserver with Master: 594d6f796cae43afb1ca62740a564f50 (127.3.19.67:35667)
I20260925 09:14:16.348556 9351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:60793
I20260925 09:14:16.360555 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.19.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.19.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.19.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.125.keytab.
Entry for principal oryx/127.3.19.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.125.keytab.
I20260925 09:14:16.412542 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.125:45179
--webserver_interface=127.3.19.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.125.keytab
--principal=oryx/127.3.19.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:39883,127.3.19.125:45179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:16.750240 9813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:16.750625 9813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:16.750718 9813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:16.758384 9813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:16.758564 9813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:16.758620 9813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:16.758680 9813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:16.758733 9813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:14:16.769101 9813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:39883,127.3.19.125:45179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.125.keytab
--principal=oryx/127.3.19.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.125:45179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.19.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
W20260925 09:14:16.769506 9813 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.19.126:39883,127.3.19.125:45179'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 09:14:16.770514 9813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:16.771759 9813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:16.784055 9821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:16.784144 9819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:16.784124 9818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:16.785117 9813 server_base.cc:1034] running on GCE node
I20260925 09:14:16.786235 9813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:16.787806 9813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:16.789085 9813 hybrid_clock.cc:648] HybridClock initialized: now 1790327656788976 us; error 67 us; skew 500 ppm
I20260925 09:14:16.789569 9813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:16 dist-test-slave-l6qh krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327656, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:16.794102 9813 init.cc:377] Logged in from keytab as oryx/127.3.19.125@KRBTEST.COM (short username oryx)
I20260925 09:14:16.795678 9813 webserver.cc:460] Webserver started at http://127.3.19.125:36013/ using document root <none> and password file <none>
I20260925 09:14:16.796422 9813 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:16.796564 9813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:16.796898 9813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:16.800091 9813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/data/instance:
uuid: "36fc01d8a2804f92b8ccacfe2302230f"
format_stamp: "Formatted at 2026-09-25 09:14:16 on dist-test-slave-l6qh"
server_key: "73788b7c4dddcbe99b2e744279a359ad"
server_key_iv: "943331337f1661418ef017caac698fa7"
server_key_version: "encryptionkey@0"
I20260925 09:14:16.801017 9813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/wal/instance:
uuid: "36fc01d8a2804f92b8ccacfe2302230f"
format_stamp: "Formatted at 2026-09-25 09:14:16 on dist-test-slave-l6qh"
server_key: "73788b7c4dddcbe99b2e744279a359ad"
server_key_iv: "943331337f1661418ef017caac698fa7"
server_key_version: "encryptionkey@0"
I20260925 09:14:16.806751 9813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 09:14:16.810539 9828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:16.812142 9813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:14:16.812455 9813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/wal
uuid: "36fc01d8a2804f92b8ccacfe2302230f"
format_stamp: "Formatted at 2026-09-25 09:14:16 on dist-test-slave-l6qh"
server_key: "73788b7c4dddcbe99b2e744279a359ad"
server_key_iv: "943331337f1661418ef017caac698fa7"
server_key_version: "encryptionkey@0"
I20260925 09:14:16.812723 9813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:16.834807 9674 heartbeater.cc:503] Master 127.3.19.126:39883 was elected leader, sending a full tablet report...
I20260925 09:14:16.856748 9813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:16.861043 9813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:16.861333 9813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:16.880416 9813 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.125:45179
I20260925 09:14:16.880439 9879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.125:45179 every 8 connection(s)
I20260925 09:14:16.881887 9813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/data/info.pb
I20260925 09:14:16.885423 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 9813
I20260925 09:14:16.887184 9880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:16.895763 9880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:14:16 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.19.125@KRBTEST.COM
Sep 25 09:14:16 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327656, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.125@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:16.910523 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:16.887958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.3.19.125:45179)
0925 09:14:16.888791 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:16.888803 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:16.888843 (+ 40us) server_negotiation.cc:407] Connection header received
0925 09:14:16.890441 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:16.890463 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:16.890871 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:16.891280 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:16.892785 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.894174 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.895406 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.896116 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.896345 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:16.901559 (+ 5214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:16.901569 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:16.901582 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:16.901633 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:16.905497 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:16.906541 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:16.906548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:16.906556 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:16.906823 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:16.907326 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:16.907329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:16.907332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:16.907887 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:16.908199 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:16.908685 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 09:14:16.909085 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"spinlock_wait_cycles":27904,"thread_start_us":410,"threads_started":1}
I20260925 09:14:16.915797 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:16.898679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.3.19.126:39883)
0925 09:14:16.899287 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:16.899297 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:16.899834 (+ 537us) server_negotiation.cc:407] Connection header received
0925 09:14:16.900047 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:16.900053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:16.900136 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:16.900223 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:16.901545 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.902672 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.904572 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.905319 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.905479 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:16.909602 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:16.909610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:16.909615 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:16.909655 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:16.912687 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:16.913711 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:16.913716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:16.913719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:16.913857 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:16.914276 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:16.914281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:16.914284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:16.914592 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:16.914757 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:16.915191 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 09:14:16.915417 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":299,"threads_started":1}
I20260925 09:14:16.915836 9885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:16.898705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:44362)
0925 09:14:16.899640 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:16.899677 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:16.899906 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:16.900341 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:16.900349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:16.901024 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:16.901384 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:16.901396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.902824 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.902828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:16.904384 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:16.904398 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.905595 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.905606 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:16.905844 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:16.906576 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:16.906595 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:16.909406 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:16.912823 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:16.912832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:16.912840 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:16.913553 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:16.913998 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:16.914003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:16.914007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:16.914149 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:16.914721 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:16.914729 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:16.914898 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 09:14:16.915094 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 09:14:16.915449 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":559,"threads_started":1}
I20260925 09:14:16.917575 9880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:14:16 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327656, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.125@KRBTEST.COM for oryx/127.3.19.125@KRBTEST.COM
I20260925 09:14:16.930037 9885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:16.918267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:45179 (local address 127.0.0.1:55476)
0925 09:14:16.918434 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:16.918453 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:16.918531 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:16.918893 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:16.918896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:16.919229 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:16.919356 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:16.919363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.920334 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.920338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:16.921439 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:16.921450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.922206 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.922211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:16.922334 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:16.922886 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:16.922902 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:16.924889 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:16.927802 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:16.927806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:16.927809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:16.928166 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:16.928518 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:16.928522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:16.928524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:16.928648 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:16.929294 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:16.929298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:16.929422 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 09:14:16.929536 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 09:14:16.929770 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 09:14:16.930276 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:16.918499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.3.19.125:45179)
0925 09:14:16.918664 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:16.918668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:16.918684 (+ 16us) server_negotiation.cc:407] Connection header received
0925 09:14:16.918724 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:16.918728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:16.918780 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:16.918863 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:16.919476 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.920193 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.921586 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.922041 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.922147 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:16.925055 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:16.925060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:16.925063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:16.925093 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:16.927673 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:16.928264 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:16.928268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:16.928270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:16.928411 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:16.928792 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:16.928801 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:16.928808 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:16.929141 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:16.929346 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:16.929668 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 09:14:16.929902 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":55}
I20260925 09:14:16.934979 9880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: Bootstrap starting.
I20260925 09:14:16.938143 9880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:16.939232 9880 log.cc:824] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:16.942078 9880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: No bootstrap required, opened a new log
I20260925 09:14:16.948916 9880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } }
I20260925 09:14:16.949282 9880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:16.949414 9880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc01d8a2804f92b8ccacfe2302230f, State: Initialized, Role: FOLLOWER
I20260925 09:14:16.950086 9880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } }
I20260925 09:14:16.950995 9887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } } }
I20260925 09:14:16.951282 9887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:14:16.952028 9880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: configured and running, proceeding with master startup.
W20260925 09:14:16.962325 9898 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 09:14:16.962515 9898 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 09:14:16.963541 9813 master_runner.cc:428] Detected that this master 36fc01d8a2804f92b8ccacfe2302230f is joining an existing cluster
I20260925 09:14:16.963644 9813 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.19.125:45179
WARNING: no policy specified for oryx/127.3.19.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.19.124@KRBTEST.COM" created.
I20260925 09:14:16.995217 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:16.968947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.3.19.125:45179)
0925 09:14:16.969148 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:16.969156 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:16.975732 (+ 6576us) server_negotiation.cc:407] Connection header received
0925 09:14:16.976040 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:16.976047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:16.976128 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:16.976198 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:16.977774 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.979827 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.982030 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.982871 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.982989 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:16.987040 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:16.987048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:16.987053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:16.987092 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:16.991758 (+ 4666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:16.992717 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:16.992724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:16.992727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:16.992879 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:16.993395 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:16.993400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:16.993404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:16.993806 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:16.993975 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:16.994497 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 09:14:16.994755 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 09:14:16.996663 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:16.968870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:45179 (local address 127.0.0.1:55478)
0925 09:14:16.975610 (+ 6740us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:16.975635 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:16.975864 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:16.976404 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:16.976411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:16.977100 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:16.977586 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:16.977604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.980028 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.980036 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:16.981868 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:16.981886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.983385 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.983397 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:16.983638 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:16.985434 (+ 1796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:16.985464 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:16.986830 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:16.991923 (+ 5093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:16.991929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:16.991933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:16.992544 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:16.993039 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:16.993044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:16.993048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:16.993212 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:16.994054 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:16.994058 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:16.994211 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 09:14:16.995820 (+ 1609us) client_negotiation.cc:240] Negotiation successful
0925 09:14:16.996142 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6561,"thread_start_us":13188,"threads_started":1}
I20260925 09:14:16.999166 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:16.968170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.3.19.126:39883)
0925 09:14:16.968366 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:16.968374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:16.982022 (+ 13648us) server_negotiation.cc:407] Connection header received
0925 09:14:16.982552 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:16.982558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:16.982668 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:16.982749 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:16.984355 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.985658 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.987983 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.988733 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.988897 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:16.991055 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:16.991061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:16.991065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:16.991106 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:16.994807 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:16.995656 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:16.995664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:16.995670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:16.995867 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:16.996373 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:16.996381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:16.996386 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:16.996761 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:16.996886 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:16.997533 (+ 647us) server_negotiation.cc:299] Negotiation successful
0925 09:14:16.998703 (+ 1170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 09:14:17.000151 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:16.974728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:44372)
0925 09:14:16.981887 (+ 7159us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:16.981913 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:16.982357 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:16.982992 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:16.982999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:16.983824 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:16.984180 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:16.984195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.985839 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.985847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:16.987808 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:16.987827 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:16.988950 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:16.988957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:16.989132 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:16.989853 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:16.989878 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:16.990849 (+ 971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:16.994963 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:16.994974 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:16.994981 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:16.995517 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:16.996040 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:16.996047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:16.996053 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:16.996232 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:16.996981 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:16.996988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:16.997214 (+ 226us) client_negotiation.cc:769] Sending connection context
0925 09:14:16.999431 (+ 2217us) client_negotiation.cc:240] Negotiation successful
0925 09:14:16.999822 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6999,"thread_start_us":949,"threads_started":1}
Sep 25 09:14:17 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327654, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.126@KRBTEST.COM for oryx/127.3.19.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.19.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.124.keytab.
Entry for principal oryx/127.3.19.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.124.keytab.
I20260925 09:14:17.027384 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.124:38283
--webserver_interface=127.3.19.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.124.keytab
--principal=oryx/127.3.19.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:39883,127.3.19.125:45179,127.3.19.124:38283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:14:17.028242 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.007469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.3.19.125:45179)
0925 09:14:17.007673 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:17.007680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:17.008445 (+ 765us) server_negotiation.cc:407] Connection header received
0925 09:14:17.008883 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:17.008891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:17.009029 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:17.009146 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:17.010794 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.012113 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.014466 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.015216 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.015460 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.021285 (+ 5825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:17.021298 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:17.021302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:17.021336 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:17.024431 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.025530 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.025536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.025539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.025711 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.026217 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.026222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.026226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.026681 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:17.026854 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:17.027489 (+ 635us) server_negotiation.cc:299] Negotiation successful
0925 09:14:17.027764 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 09:14:17.028251 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.006506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:45179 (local address 127.0.0.1:55484)
0925 09:14:17.008291 (+ 1785us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.008337 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:17.008605 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:17.009271 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:17.009290 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:17.010244 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:17.010596 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.010611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.012299 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.012306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.014256 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.014275 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.015451 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.015460 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.015705 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.016558 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:17.016582 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:17.020998 (+ 4416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:17.024624 (+ 3626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.024642 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.024652 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.025356 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.025853 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.025859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.025863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.026061 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.026868 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:17.026878 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:17.027107 (+ 229us) client_negotiation.cc:769] Sending connection context
0925 09:14:17.027333 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 09:14:17.027748 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1496,"thread_start_us":1314,"threads_started":1}
I20260925 09:14:17.030645 9351 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.19.125:45179: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f"
member_type: NON_VOTER
last_known_addr {
host: "127.3.19.125"
port: 45179
}
attrs {
promote: true
}
}
dest_uuid: "3cd59f740532408ba0b9dd8b2012096a"
cas_config_opid_index: -1
I20260925 09:14:17.031873 9351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true } }
I20260925 09:14:17.040427 9911 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 36fc01d8a2804f92b8ccacfe2302230f (127.3.19.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true } } }
I20260925 09:14:17.041707 9855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:17.045389 9911 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.19.125:45179
I20260925 09:14:17.045526 9855 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 FOLLOWER]: Refusing update from remote peer 3cd59f740532408ba0b9dd8b2012096a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 09:14:17.046507 9910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true } } }
I20260925 09:14:17.046792 9813 master.cc:537] Master@127.3.19.125:45179 shutting down...
I20260925 09:14:17.047971 9910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:17.050040 9911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:17.062526 9813 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:17.063349 9813 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:17.064071 9813 tablet_replica.cc:330] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: stopping tablet replica
I20260925 09:14:17.075035 9813 master.cc:552] Master@127.3.19.125:45179 shutdown complete.
I20260925 09:14:17.076404 9813 master_runner.cc:305] Clearing existing system tablet
I20260925 09:14:17.076997 9813 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:17.077623 9813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:17.083630 9813 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 09:14:17.085700 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.085114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:45179 (local address 127.0.0.1:55492)
0925 09:14:17.085345 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.085499 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:45179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 09:14:17.085986 9336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.19.125:45179: connect: Connection refused (error 111)
I20260925 09:14:17.085748 9916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:17.087767 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.087315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:45179 (local address 127.0.0.1:55504)
0925 09:14:17.087501 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.087604 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:45179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 09:14:17.087697 9813 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:17.087967 9813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:14:17.088054 9813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/wal
uuid: "36fc01d8a2804f92b8ccacfe2302230f"
format_stamp: "Formatted at 2026-09-25 09:14:16 on dist-test-slave-l6qh"
server_key: "73788b7c4dddcbe99b2e744279a359ad"
server_key_iv: "943331337f1661418ef017caac698fa7"
server_key_version: "encryptionkey@0"
W20260925 09:14:17.088160 9336 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a -> Peer 36fc01d8a2804f92b8ccacfe2302230f (127.3.19.125:45179): Couldn't send request to peer 36fc01d8a2804f92b8ccacfe2302230f. Status: Network error: Client connection negotiation failed: client connection to 127.3.19.125:45179: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 09:14:17.092634 9813 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:17.099622 9813 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 09:14:17.099815 9813 log.cc:1196] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 09:14:17.100289 9813 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: Deleting consensus metadata
I20260925 09:14:17.100558 9813 master_runner.cc:315] Copying system tablet from 127.3.19.126:39883
I20260925 09:14:17.103072 9813 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: tablet copy: Beginning tablet copy session from remote peer at address 127.3.19.126:39883
I20260925 09:14:17.120358 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.104012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.3.19.126:39883)
0925 09:14:17.104202 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:17.104210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:17.104746 (+ 536us) server_negotiation.cc:407] Connection header received
0925 09:14:17.104937 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:17.104943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:17.105029 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:17.105141 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:17.106478 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.107613 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.109495 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.110179 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.110326 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.113303 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:17.113310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:17.113316 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:17.113360 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:17.117064 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.118042 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.118047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.118050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.118206 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.118688 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.118693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.118696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.119094 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:17.119267 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:17.119766 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 09:14:17.120005 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 09:14:17.120409 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.103874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:44376)
0925 09:14:17.104593 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.104619 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:17.104788 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:17.105207 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:17.105212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:17.105855 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:17.106220 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.106245 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.107777 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.107782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.109337 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.109351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.110357 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.110362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.110561 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.111291 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:17.111314 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:17.112485 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:17.117299 (+ 4814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.117308 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.117315 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.117832 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.118339 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.118345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.118349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.118541 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.119250 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:17.119256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:17.119470 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 09:14:17.119624 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 09:14:17.119987 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":473,"threads_started":1}
I20260925 09:14:17.121658 9371 tablet_copy_service.cc:140] P 3cd59f740532408ba0b9dd8b2012096a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 36fc01d8a2804f92b8ccacfe2302230f ({username='oryx', principal='oryx/127.3.19.125@KRBTEST.COM'} at 127.0.0.1:44376)
I20260925 09:14:17.121948 9371 tablet_copy_service.cc:161] P 3cd59f740532408ba0b9dd8b2012096a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 36fc01d8a2804f92b8ccacfe2302230f at {username='oryx', principal='oryx/127.3.19.125@KRBTEST.COM'} at 127.0.0.1:44376: session id = 36fc01d8a2804f92b8ccacfe2302230f-00000000000000000000000000000000
I20260925 09:14:17.126318 9371 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 09:14:17.128763 9813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 09:14:17.135762 9813 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: tablet copy: Starting download of 0 data blocks...
I20260925 09:14:17.136087 9813 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: tablet copy: Starting download of 1 WAL segments...
I20260925 09:14:17.139070 9813 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 09:14:17.143836 9371 tablet_copy_service.cc:342] P 3cd59f740532408ba0b9dd8b2012096a: Request end of tablet copy session 36fc01d8a2804f92b8ccacfe2302230f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.19.125@KRBTEST.COM'} at 127.0.0.1:44376
I20260925 09:14:17.144078 9371 tablet_copy_service.cc:433] P 3cd59f740532408ba0b9dd8b2012096a: ending tablet copy session 36fc01d8a2804f92b8ccacfe2302230f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 36fc01d8a2804f92b8ccacfe2302230f
I20260925 09:14:17.155995 9813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:17.159075 9924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:17.169036 9927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:17.169286 9925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:17.169760 9813 server_base.cc:1034] running on GCE node
I20260925 09:14:17.170089 9813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:17.170729 9813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:17.172034 9813 hybrid_clock.cc:648] HybridClock initialized: now 1790327657171961 us; error 47 us; skew 500 ppm
I20260925 09:14:17.172260 9813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:17 dist-test-slave-l6qh krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327657, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:17.174181 9813 init.cc:377] Logged in from keytab as oryx/127.3.19.125@KRBTEST.COM (short username oryx)
I20260925 09:14:17.175413 9813 webserver.cc:460] Webserver started at http://127.3.19.125:42217/ using document root <none> and password file <none>
I20260925 09:14:17.175740 9813 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:17.175833 9813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:17.178126 9813 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260925 09:14:17.180043 9934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:17.180516 9813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 09:14:17.180676 9813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/wal
uuid: "36fc01d8a2804f92b8ccacfe2302230f"
format_stamp: "Formatted at 2026-09-25 09:14:16 on dist-test-slave-l6qh"
server_key: "73788b7c4dddcbe99b2e744279a359ad"
server_key_iv: "943331337f1661418ef017caac698fa7"
server_key_version: "encryptionkey@0"
I20260925 09:14:17.180846 9813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:17.189889 9813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:17.193290 9813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:17.222277 9813 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.125:45179
I20260925 09:14:17.222292 9985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.125:45179 every 8 connection(s)
I20260925 09:14:17.223245 9813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-1/data/info.pb
I20260925 09:14:17.230566 9986 sys_catalog.cc:263] Verifying existing consensus state
I20260925 09:14:17.236033 9986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: Bootstrap starting.
I20260925 09:14:17.249398 9986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 09:14:17.249908 9986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: Bootstrap complete.
I20260925 09:14:17.250999 9986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true } }
I20260925 09:14:17.251245 9986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 36fc01d8a2804f92b8ccacfe2302230f, State: Initialized, Role: LEARNER
I20260925 09:14:17.251495 9986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true } }
I20260925 09:14:17.252321 9989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true } } }
I20260925 09:14:17.252633 9986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:17.252707 9989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: This master's current role is: LEARNER
I20260925 09:14:17.273658 10000 catalog_manager.cc:1259] Loaded cluster ID: c0376b7b833547c7924a8ec1c3637e51
I20260925 09:14:17.273797 10000 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: loading cluster ID for follower catalog manager: success
I20260925 09:14:17.276216 10000 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: acquiring CA information for follower catalog manager: success
I20260925 09:14:17.278162 10000 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 09:14:17.351639 9808 heartbeater.cc:503] Master 127.3.19.126:39883 was elected leader, sending a full tablet report...
I20260925 09:14:17.517297 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.508378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:45179 (local address 127.0.0.1:55520)
0925 09:14:17.508645 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.508666 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:17.508804 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:17.509781 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:17.509786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:17.509820 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:17.510006 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.510016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.512420 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.512427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.514492 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.514510 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.516286 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.516293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.516526 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.516611 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 09:14:17.516729 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 09:14:17.516916 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 09:14:17.517346 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.508661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.3.19.125:45179)
0925 09:14:17.509372 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:17.509378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:17.509398 (+ 20us) server_negotiation.cc:407] Connection header received
0925 09:14:17.509468 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:17.509473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:17.509560 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:17.509691 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:17.510189 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.512229 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.514677 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.516088 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.516227 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.516474 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:17.516769 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 09:14:17.516924 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":385,"threads_started":1}
W20260925 09:14:17.517342 9912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:17.517783 9912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:17.517967 9912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 09:14:17.518970 9961 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 09:14:17.519836 9989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cd59f740532408ba0b9dd8b2012096a. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true } } }
I20260925 09:14:17.520109 9989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: This master's current role is: LEARNER
W20260925 09:14:17.526641 9912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:17.526751 9912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:17.526882 9912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:17.526983 9912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:17.527067 9912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:14:17.538187 9912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:39883,127.3.19.125:45179,127.3.19.124:38283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.124.keytab
--principal=oryx/127.3.19.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.124:38283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.19.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:17.539673 9912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:17.541272 9912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:17.554399 10006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:17.554416 10009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:17.554360 10007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:17.555783 9912 server_base.cc:1034] running on GCE node
I20260925 09:14:17.556670 9912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:17.558327 9912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:17.559621 9912 hybrid_clock.cc:648] HybridClock initialized: now 1790327657559494 us; error 68 us; skew 500 ppm
I20260925 09:14:17.560042 9912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:17 dist-test-slave-l6qh krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327657, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:17.563464 9912 init.cc:377] Logged in from keytab as oryx/127.3.19.124@KRBTEST.COM (short username oryx)
I20260925 09:14:17.564973 9912 webserver.cc:460] Webserver started at http://127.3.19.124:37485/ using document root <none> and password file <none>
I20260925 09:14:17.565764 9912 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:17.565868 9912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:17.566116 9912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:17.569249 9912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/data/instance:
uuid: "1ffbdef9809940d5bf56c324e26bd133"
format_stamp: "Formatted at 2026-09-25 09:14:17 on dist-test-slave-l6qh"
server_key: "71208ccc57765e425beff83a16f20968"
server_key_iv: "07ce804bf68f67a7c9c0ffe475f0e9c2"
server_key_version: "encryptionkey@0"
I20260925 09:14:17.570125 9912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/wal/instance:
uuid: "1ffbdef9809940d5bf56c324e26bd133"
format_stamp: "Formatted at 2026-09-25 09:14:17 on dist-test-slave-l6qh"
server_key: "71208ccc57765e425beff83a16f20968"
server_key_iv: "07ce804bf68f67a7c9c0ffe475f0e9c2"
server_key_version: "encryptionkey@0"
I20260925 09:14:17.576118 9912 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 09:14:17.579991 10016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:17.581709 9912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:17.581918 9912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/wal
uuid: "1ffbdef9809940d5bf56c324e26bd133"
format_stamp: "Formatted at 2026-09-25 09:14:17 on dist-test-slave-l6qh"
server_key: "71208ccc57765e425beff83a16f20968"
server_key_iv: "07ce804bf68f67a7c9c0ffe475f0e9c2"
server_key_version: "encryptionkey@0"
I20260925 09:14:17.582180 9912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:17.601344 9912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:17.605894 9912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:17.606213 9912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:17.629180 9912 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.124:38283
I20260925 09:14:17.629206 10067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.124:38283 every 8 connection(s)
I20260925 09:14:17.630756 9912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/data/info.pb
I20260925 09:14:17.634564 10068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:17.638336 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 9912
I20260925 09:14:17.642103 10068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:14:17 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327657, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.124@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
Sep 25 09:14:17 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.19.124@KRBTEST.COM
I20260925 09:14:17.666577 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.646164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.3.19.126:39883)
0925 09:14:17.646647 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:17.646652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:17.647519 (+ 867us) server_negotiation.cc:407] Connection header received
0925 09:14:17.647751 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:17.647757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:17.647832 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:17.647973 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:17.649483 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.650742 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.653155 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.653793 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.653935 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.658856 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:17.658863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:17.658869 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:17.658912 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:17.662775 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.664334 (+ 1559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.664339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.664341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.664494 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.664864 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.664868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.664870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.665124 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:17.665329 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:17.665879 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 09:14:17.666120 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":285,"threads_started":1}
I20260925 09:14:17.667497 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.640675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.3.19.124:38283)
0925 09:14:17.641443 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:17.641456 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:17.641504 (+ 48us) server_negotiation.cc:407] Connection header received
0925 09:14:17.643156 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:17.643180 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:17.643637 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:17.644129 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:17.645963 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.648376 (+ 2413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.650469 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.651938 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.652135 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.659682 (+ 7547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:17.659691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:17.659706 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:17.659770 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:17.662783 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.663556 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.663566 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.663577 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.663847 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.664285 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.664292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.664296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.665020 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:17.665356 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:17.665751 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 09:14:17.666243 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"spinlock_wait_cycles":1792,"thread_start_us":330,"threads_started":1}
I20260925 09:14:17.667515 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.646534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:44390)
0925 09:14:17.647311 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.647356 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:17.647609 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:17.648031 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:17.648041 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:17.648869 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:17.649296 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.649309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.650940 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.650949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.652981 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.652996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.654037 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.654045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.654283 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.655111 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:17.655144 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:17.658575 (+ 3431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:17.663556 (+ 4981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.663566 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.663580 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.664198 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.664608 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.664613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.664616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.664760 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.665297 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:17.665306 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:17.665529 (+ 223us) client_negotiation.cc:769] Sending connection context
0925 09:14:17.665788 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 09:14:17.666421 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":128,"spinlock_wait_cycles":1408,"thread_start_us":432,"threads_started":1}
I20260925 09:14:17.670075 10068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:14:17 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327657, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.124@KRBTEST.COM for oryx/127.3.19.125@KRBTEST.COM
I20260925 09:14:17.683246 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.670993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.3.19.125:45179)
0925 09:14:17.671175 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:17.671180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:17.671197 (+ 17us) server_negotiation.cc:407] Connection header received
0925 09:14:17.671265 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:17.671269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:17.671326 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:17.671405 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:17.672115 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.672817 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.674546 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.674951 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.675065 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.678186 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:17.678190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:17.678194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:17.678225 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:17.680791 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.681547 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.681551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.681553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.681673 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.682065 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.682068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.682070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.682387 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:17.682505 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:17.682828 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 09:14:17.683009 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 09:14:17.683316 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.670875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:45179 (local address 127.0.0.1:55530)
0925 09:14:17.671075 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.671100 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:17.671182 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:17.671416 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:17.671419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:17.671829 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:17.671976 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.671984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.672950 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.672954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.674390 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.674407 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.675134 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.675139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.675286 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.675869 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:17.675886 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:17.677995 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:17.680921 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.680924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.680927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.681444 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.681773 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.681776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.681779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.681955 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.682487 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:17.682491 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:17.682628 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 09:14:17.682733 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 09:14:17.682949 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 09:14:17.684460 10068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:14:17 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327657, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.124@KRBTEST.COM for oryx/127.3.19.124@KRBTEST.COM
I20260925 09:14:17.697582 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.685155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:38283 (local address 127.0.0.1:51526)
0925 09:14:17.685371 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.685392 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:17.685524 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:17.685836 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:17.685840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:17.686230 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:17.686366 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.686373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.687766 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.687775 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.688710 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.688721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.689461 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.689466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.689684 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.690295 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:17.690310 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:17.692261 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:17.695116 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.695123 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.695129 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.695552 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.695906 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.695913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.695917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.696076 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.696569 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:17.696573 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:17.696702 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 09:14:17.696851 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 09:14:17.697133 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 09:14:17.697610 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.685211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51526 (local address 127.3.19.124:38283)
0925 09:14:17.685371 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:17.685377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:17.685468 (+ 91us) server_negotiation.cc:407] Connection header received
0925 09:14:17.685642 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:17.685649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:17.685723 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:17.685823 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:17.686502 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.687634 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.688841 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.689324 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.689454 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.692448 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:17.692453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:17.692456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:17.692486 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:17.694980 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.695669 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.695677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.695681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.695801 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.696178 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.696186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.696190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.696450 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:17.696606 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:17.696920 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 09:14:17.697287 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":151,"server-negotiator.queue_time_us":77}
I20260925 09:14:17.702409 10068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: Bootstrap starting.
I20260925 09:14:17.705766 10068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:17.707000 10068 log.cc:824] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:17.710013 10068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: No bootstrap required, opened a new log
I20260925 09:14:17.717770 10068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } }
I20260925 09:14:17.718118 10068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:17.718196 10068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffbdef9809940d5bf56c324e26bd133, State: Initialized, Role: FOLLOWER
I20260925 09:14:17.718719 10068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } }
I20260925 09:14:17.719732 10075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } } }
I20260925 09:14:17.720116 10075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:14:17.720901 10068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 09:14:17.731317 10086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 09:14:17.731482 10086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 09:14:17.736063 9912 master_runner.cc:428] Detected that this master 1ffbdef9809940d5bf56c324e26bd133 is joining an existing cluster
I20260925 09:14:17.736187 9912 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.19.124:38283
I20260925 09:14:17.760833 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.740415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.3.19.126:39883)
0925 09:14:17.740649 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:17.740657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:17.742688 (+ 2031us) server_negotiation.cc:407] Connection header received
0925 09:14:17.743201 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:17.743210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:17.743314 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:17.743393 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:17.745733 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.747194 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.749788 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.750494 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.750603 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.753500 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:17.753507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:17.753513 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:17.753555 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:17.757627 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.758460 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.758467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.758471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.758628 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.759146 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.759151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.759154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.759593 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:17.759747 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:17.760236 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 09:14:17.760472 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 09:14:17.760993 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.740878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:44402)
0925 09:14:17.742567 (+ 1689us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.742587 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:17.743000 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:17.743644 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:17.743652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:17.744649 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:17.745468 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.745488 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.747466 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.747474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.749628 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.749644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.750909 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.750915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.751108 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.751952 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:17.751976 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:17.753280 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:17.757755 (+ 4475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.757759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.757764 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.758284 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.758750 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.758755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.758758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.759005 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.759756 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:17.759763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:17.759962 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 09:14:17.760125 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 09:14:17.760465 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1561,"mutex_wait_us":1,"thread_start_us":1182,"threads_started":1}
I20260925 09:14:17.774093 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 9412
Sep 25 09:14:17 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.19.65@KRBTEST.COM
I20260925 09:14:17.788468 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.742563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51530 (local address 127.3.19.124:38283)
0925 09:14:17.742740 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:17.742746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:17.749646 (+ 6900us) server_negotiation.cc:407] Connection header received
0925 09:14:17.750007 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:17.750011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:17.750070 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:17.750132 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:17.751780 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.752638 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.755502 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.756171 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.756345 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.758454 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:17.758462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:17.758468 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:17.758500 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:17.762592 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.786034 (+ 23442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.786039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.786042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.786231 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.786642 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.786647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.786650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.787041 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:17.787175 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:17.787762 (+ 587us) server_negotiation.cc:299] Negotiation successful
0925 09:14:17.788032 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 09:14:17.789455 10094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.742863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:38283 (local address 127.0.0.1:51530)
0925 09:14:17.749548 (+ 6685us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.749568 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:17.749884 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:17.750297 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:17.750302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:17.751138 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:17.751642 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.751652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.752844 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.752850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.755363 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.755375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.756370 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.756375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.756565 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.757155 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:17.757172 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:17.758266 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:17.762723 (+ 4457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.762727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.762729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.785856 (+ 23127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.786374 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.786378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.786382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.786533 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.787243 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:17.787249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:17.787452 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 09:14:17.788648 (+ 1196us) client_negotiation.cc:240] Negotiation successful
0925 09:14:17.788954 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6492,"thread_start_us":6800,"threads_started":1}
I20260925 09:14:17.797653 10092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.740844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:45179 (local address 127.0.0.1:55544)
0925 09:14:17.741657 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.741686 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:17.741843 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:17.742259 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:17.742264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:17.743094 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:17.743476 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.743489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.745419 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.745430 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.748284 (+ 2854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.748299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.749550 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.749558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.749746 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.750557 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:17.750584 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:17.790732 (+ 40148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:17.794748 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.794754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.794758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.795233 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.795681 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.795685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.795689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.795860 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.796509 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:17.796514 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:17.796709 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 09:14:17.796854 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 09:14:17.797166 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":440,"threads_started":1}
I20260925 09:14:17.797946 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.775275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.3.19.65:39165)
0925 09:14:17.776457 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:17.776465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:17.776494 (+ 29us) server_negotiation.cc:407] Connection header received
0925 09:14:17.776553 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:17.776559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:17.776860 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:17.777015 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:17.778778 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.779981 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.781891 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.782632 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.782741 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.788758 (+ 6017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:17.788778 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:17.788798 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:17.788842 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:17.793572 (+ 4730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.794521 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.794531 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.794535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.794690 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.795152 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.795158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.795160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.795583 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:17.797060 (+ 1477us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:17.797248 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 09:14:17.797517 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":842,"threads_started":1}
I20260925 09:14:17.799676 9535 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 25 09:14:17 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327654, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.126@KRBTEST.COM for oryx/127.3.19.124@KRBTEST.COM
I20260925 09:14:17.802686 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.740511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.3.19.125:45179)
0925 09:14:17.740734 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:17.740742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:17.741805 (+ 1063us) server_negotiation.cc:407] Connection header received
0925 09:14:17.741990 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:17.741999 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:17.742099 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:17.742204 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:17.743669 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.745204 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.748464 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.749126 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.749274 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.791001 (+ 41727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:17.791008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:17.791014 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:17.791055 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:17.794574 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.795394 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.795400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.795404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.795555 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.795992 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.795996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.795999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.796319 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:17.796449 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:17.798027 (+ 1578us) server_negotiation.cc:299] Negotiation successful
0925 09:14:17.798269 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 09:14:17.812079 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.791051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:38283 (local address 127.0.0.1:51544)
0925 09:14:17.791543 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.791569 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:17.791699 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:17.793803 (+ 2104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:17.793808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:17.794316 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:17.794566 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.794577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.796154 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.796161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.797556 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.797570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.799099 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.799105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.799347 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.800073 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:17.800094 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:17.803229 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:17.809054 (+ 5825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.809061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.809066 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.809659 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.810100 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.810106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.810110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.810259 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.810972 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:17.810978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:17.811148 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 09:14:17.811297 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 09:14:17.811581 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260925 09:14:17.812090 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.791932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.3.19.124:38283)
0925 09:14:17.793376 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:17.793383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:17.793399 (+ 16us) server_negotiation.cc:407] Connection header received
0925 09:14:17.793508 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:17.793514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:17.793628 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:17.794711 (+ 1083us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:17.794904 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.796013 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.798246 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.798924 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.803537 (+ 4613us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.803761 (+ 224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:17.803769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:17.803777 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:17.803811 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:17.808850 (+ 5039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.809808 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.809814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.809818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.809966 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.810404 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.810417 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.810420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.810832 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:17.810984 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:17.811457 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 09:14:17.811708 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1322}
I20260925 09:14:17.814348 9351 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.19.124:38283: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133"
member_type: NON_VOTER
last_known_addr {
host: "127.3.19.124"
port: 38283
}
attrs {
promote: true
}
}
dest_uuid: "3cd59f740532408ba0b9dd8b2012096a"
cas_config_opid_index: 5
I20260925 09:14:17.815419 9351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } }
I20260925 09:14:17.817148 9910 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1ffbdef9809940d5bf56c324e26bd133 (127.3.19.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } } }
I20260925 09:14:17.818235 10043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:17.822038 10043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 FOLLOWER]: Refusing update from remote peer 3cd59f740532408ba0b9dd8b2012096a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 09:14:17.822023 9961 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 LEARNER]: Refusing update from remote peer 3cd59f740532408ba0b9dd8b2012096a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 09:14:17.822431 9910 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.19.124:38283
I20260925 09:14:17.823050 9912 master.cc:537] Master@127.3.19.124:38283 shutting down...
I20260925 09:14:17.824481 9910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 09:14:17.825081 10097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } } }
I20260925 09:14:17.825609 10097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:17.842039 9912 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:17.842497 9912 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:17.842697 9912 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: stopping tablet replica
I20260925 09:14:17.850651 10098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:17.855768 9961 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1ffbdef9809940d5bf56c324e26bd133 (127.3.19.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } } }
I20260925 09:14:17.861896 9912 master.cc:552] Master@127.3.19.124:38283 shutdown complete.
I20260925 09:14:17.862006 9912 master_runner.cc:305] Clearing existing system tablet
I20260925 09:14:17.861829 9989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: NON_VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: true } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } } }
I20260925 09:14:17.862187 9989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: This master's current role is: LEARNER
I20260925 09:14:17.862427 9912 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:17.862538 9912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:17.873797 9910 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a: attempting to promote NON_VOTER 36fc01d8a2804f92b8ccacfe2302230f to VOTER
I20260925 09:14:17.876760 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.876045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:38283 (local address 127.0.0.1:51550)
0925 09:14:17.876371 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.876503 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260925 09:14:17.878248 9910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } }
I20260925 09:14:17.878652 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.878155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:38283 (local address 127.0.0.1:51558)
0925 09:14:17.878387 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.878482 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260925 09:14:17.881520 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.880821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:38283 (local address 127.0.0.1:51572)
0925 09:14:17.881137 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.881311 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260925 09:14:17.882666 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.882139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:38283 (local address 127.0.0.1:51580)
0925 09:14:17.882390 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.882518 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260925 09:14:17.882704 9961 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 LEARNER]: Refusing update from remote peer 3cd59f740532408ba0b9dd8b2012096a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260925 09:14:17.882997 9338 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a -> Peer 1ffbdef9809940d5bf56c324e26bd133 (127.3.19.124:38283): Couldn't send request to peer 1ffbdef9809940d5bf56c324e26bd133. Status: Network error: Client connection negotiation failed: client connection to 127.3.19.124:38283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 09:14:17.883337 10101 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 09:14:17.887709 9961 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 36fc01d8a2804f92b8ccacfe2302230f (127.3.19.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } } }
I20260925 09:14:17.893621 9989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } } }
I20260925 09:14:17.894009 9989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:14:17.898370 10097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 36fc01d8a2804f92b8ccacfe2302230f (127.3.19.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } } }
I20260925 09:14:17.903450 9912 fs_manager.cc:709] Time spent opening directory manager: real 0.040s user 0.021s sys 0.020s
I20260925 09:14:17.903679 10097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } } }
I20260925 09:14:17.904124 10097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:17.904197 10101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } } }
I20260925 09:14:17.904471 10101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:17.905606 10106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:17.905992 9912 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:17.906244 9912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 09:14:17.906340 9912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/wal
uuid: "1ffbdef9809940d5bf56c324e26bd133"
format_stamp: "Formatted at 2026-09-25 09:14:17 on dist-test-slave-l6qh"
server_key: "71208ccc57765e425beff83a16f20968"
server_key_iv: "07ce804bf68f67a7c9c0ffe475f0e9c2"
server_key_version: "encryptionkey@0"
I20260925 09:14:17.911514 9912 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:17.919371 9912 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 09:14:17.919581 9912 log.cc:1196] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 09:14:17.920104 9912 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: Deleting consensus metadata
I20260925 09:14:17.920377 9912 master_runner.cc:315] Copying system tablet from 127.3.19.126:39883
I20260925 09:14:17.923871 9912 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: tablet copy: Beginning tablet copy session from remote peer at address 127.3.19.126:39883
I20260925 09:14:17.941886 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.924834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44418 (local address 127.3.19.126:39883)
0925 09:14:17.926133 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:17.926140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:17.926683 (+ 543us) server_negotiation.cc:407] Connection header received
0925 09:14:17.926964 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:17.926970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:17.927050 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:17.927118 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:17.928357 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.929626 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.931539 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.932258 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.932399 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.934959 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:17.934965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:17.934970 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:17.935015 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:17.938727 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.939634 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.939640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.939644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.939806 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:17.940229 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:17.940235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:17.940238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:17.940572 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:17.940709 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:17.941280 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 09:14:17.941516 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1167}
I20260925 09:14:17.942236 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:17.924839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:44418)
0925 09:14:17.926550 (+ 1711us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:17.926575 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:17.926803 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:17.927325 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:17.927329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:17.927890 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:17.928165 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.928181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.929815 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.929821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.931342 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:17.931359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:17.932441 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:17.932448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:17.932640 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:17.933551 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:17.933573 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:17.934708 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:17.938895 (+ 4187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.938901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.938906 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.939486 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.939934 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:17.939939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:17.939943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:17.940117 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:17.940695 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:17.940703 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:17.940905 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 09:14:17.941067 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 09:14:17.941449 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1535,"thread_start_us":1404,"threads_started":1}
I20260925 09:14:17.943056 9371 tablet_copy_service.cc:140] P 3cd59f740532408ba0b9dd8b2012096a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1ffbdef9809940d5bf56c324e26bd133 ({username='oryx', principal='oryx/127.3.19.124@KRBTEST.COM'} at 127.0.0.1:44418)
I20260925 09:14:17.943226 9371 tablet_copy_service.cc:161] P 3cd59f740532408ba0b9dd8b2012096a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1ffbdef9809940d5bf56c324e26bd133 at {username='oryx', principal='oryx/127.3.19.124@KRBTEST.COM'} at 127.0.0.1:44418: session id = 1ffbdef9809940d5bf56c324e26bd133-00000000000000000000000000000000
I20260925 09:14:17.946882 9371 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 09:14:17.948908 9912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 09:14:17.956028 9912 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: tablet copy: Starting download of 0 data blocks...
I20260925 09:14:17.956480 9912 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: tablet copy: Starting download of 1 WAL segments...
I20260925 09:14:17.959435 9912 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 09:14:17.964201 9371 tablet_copy_service.cc:342] P 3cd59f740532408ba0b9dd8b2012096a: Request end of tablet copy session 1ffbdef9809940d5bf56c324e26bd133-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.19.124@KRBTEST.COM'} at 127.0.0.1:44418
I20260925 09:14:17.964362 9371 tablet_copy_service.cc:433] P 3cd59f740532408ba0b9dd8b2012096a: ending tablet copy session 1ffbdef9809940d5bf56c324e26bd133-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1ffbdef9809940d5bf56c324e26bd133
I20260925 09:14:17.973887 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 9412
I20260925 09:14:17.978369 9912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:17.982592 10113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:17.982885 10114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:17.987473 10117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:17.988011 9912 server_base.cc:1034] running on GCE node
I20260925 09:14:17.988468 9912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:17.989145 9912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:17.990343 9912 hybrid_clock.cc:648] HybridClock initialized: now 1790327657990304 us; error 33 us; skew 500 ppm
I20260925 09:14:17.990590 9912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:17 dist-test-slave-l6qh krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327657, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:17.992882 9912 init.cc:377] Logged in from keytab as oryx/127.3.19.124@KRBTEST.COM (short username oryx)
I20260925 09:14:17.998025 9912 webserver.cc:460] Webserver started at http://127.3.19.124:38159/ using document root <none> and password file <none>
I20260925 09:14:17.998281 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:39165
--local_ip_for_outbound_sockets=127.3.19.65
--tserver_master_addrs=127.3.19.126:39883
--webserver_port=44921
--webserver_interface=127.3.19.65
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab
--principal=oryx/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.19.126:39883,127.3.19.125:45179
--tserver_master_addrs=127.3.19.126:39883,127.3.19.125:45179,127.3.19.124:38283 with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:14:17.998353 9912 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:17.998449 9912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:18.001691 9912 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 09:14:18.003759 10123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:18.004369 9912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 09:14:18.004526 9912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/wal
uuid: "1ffbdef9809940d5bf56c324e26bd133"
format_stamp: "Formatted at 2026-09-25 09:14:17 on dist-test-slave-l6qh"
server_key: "71208ccc57765e425beff83a16f20968"
server_key_iv: "07ce804bf68f67a7c9c0ffe475f0e9c2"
server_key_version: "encryptionkey@0"
I20260925 09:14:18.004709 9912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:18.049175 9912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:18.053697 9912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:18.089946 9912 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.124:38283
I20260925 09:14:18.090000 10175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.124:38283 every 8 connection(s)
I20260925 09:14:18.091089 9912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/master-2/data/info.pb
I20260925 09:14:18.096815 10176 sys_catalog.cc:263] Verifying existing consensus state
I20260925 09:14:18.100801 10176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: Bootstrap starting.
I20260925 09:14:18.120296 10176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 09:14:18.120842 10176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: Bootstrap complete.
I20260925 09:14:18.122040 10176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } }
I20260925 09:14:18.122351 10176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ffbdef9809940d5bf56c324e26bd133, State: Initialized, Role: LEARNER
I20260925 09:14:18.122625 10176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } }
I20260925 09:14:18.123571 10179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } } }
I20260925 09:14:18.124140 10179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: This master's current role is: LEARNER
I20260925 09:14:18.123957 10176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:18.133337 10190 catalog_manager.cc:1259] Loaded cluster ID: c0376b7b833547c7924a8ec1c3637e51
I20260925 09:14:18.133461 10190 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: loading cluster ID for follower catalog manager: success
I20260925 09:14:18.136108 10190 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: acquiring CA information for follower catalog manager: success
I20260925 09:14:18.138048 10190 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 09:14:18.272392 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:18.265227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:38283 (local address 127.0.0.1:51590)
0925 09:14:18.265768 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:18.265795 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:18.265966 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:18.267389 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:18.267393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:18.267420 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:18.267631 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:18.267641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.269063 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.269070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:18.270328 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:18.270341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.271558 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.271564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:18.271768 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:18.271816 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 09:14:18.271914 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 09:14:18.272098 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357}
I20260925 09:14:18.272441 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:18.265369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.3.19.124:38283)
0925 09:14:18.267063 (+ 1694us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:18.267067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:18.267084 (+ 17us) server_negotiation.cc:407] Connection header received
0925 09:14:18.267146 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:18.267149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:18.267217 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:18.267367 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:18.267785 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.268904 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.270508 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.271378 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.271539 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:18.271828 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:18.271935 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 09:14:18.272084 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1588,"thread_start_us":1403,"threads_started":1}
I20260925 09:14:18.273504 10151 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 09:14:18.274137 10179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cd59f740532408ba0b9dd8b2012096a. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } } }
I20260925 09:14:18.274441 10179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: This master's current role is: LEARNER
W20260925 09:14:18.414040 10124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:18.414358 10124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:18.414561 10124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:18.422405 10124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:18.422533 10124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:18.422729 10124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:14:18.433738 10124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab
--principal=oryx/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:39165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=44921
--tserver_master_addrs=127.3.19.126:39883,127.3.19.125:45179,127.3.19.124:38283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:18.435124 10124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:18.436465 10124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:18.452155 10196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:18.452095 10197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:18.452010 10199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:18.452934 10124 server_base.cc:1034] running on GCE node
I20260925 09:14:18.453647 10124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:18.454771 10124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:18.456023 10124 hybrid_clock.cc:648] HybridClock initialized: now 1790327658455967 us; error 37 us; skew 500 ppm
I20260925 09:14:18.456355 10124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:18 dist-test-slave-l6qh krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327658, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:18.459861 10124 init.cc:377] Logged in from keytab as oryx/127.3.19.65@KRBTEST.COM (short username oryx)
I20260925 09:14:18.461661 10124 webserver.cc:460] Webserver started at http://127.3.19.65:44921/ using document root <none> and password file <none>
I20260925 09:14:18.462540 10124 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:18.462735 10124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:18.468874 10124 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 09:14:18.472244 10206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:18.473731 10124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 09:14:18.473930 10124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "ff75d2f51e324151a2b0a4e0705a5fcb"
format_stamp: "Formatted at 2026-09-25 09:14:15 on dist-test-slave-l6qh"
server_key: "11e94c0634f5eabe08a438af0037dd18"
server_key_iv: "06ccf07e76936fba8a5332250262aa8b"
server_key_version: "encryptionkey@0"
I20260925 09:14:18.475032 10124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:18.499387 10124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:18.505736 10124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:18.506160 10124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:18.507413 10124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:18.509433 10124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:18.509604 10124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:18.509756 10124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:18.509884 10124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:18.570216 10124 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:39165
I20260925 09:14:18.570322 10318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:39165 every 8 connection(s)
I20260925 09:14:18.571818 10124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:14:18.574786 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 10124
I20260925 09:14:18.575258 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 9546
Sep 25 09:14:18 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327658, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.125@KRBTEST.COM
Sep 25 09:14:18 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.19.66@KRBTEST.COM
Sep 25 09:14:18 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327658, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.124@KRBTEST.COM
I20260925 09:14:18.624914 10327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:18.577411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.3.19.66:36105)
0925 09:14:18.589143 (+ 11732us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:18.589153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:18.595266 (+ 6113us) server_negotiation.cc:407] Connection header received
0925 09:14:18.595715 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:18.595724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:18.596008 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:18.596147 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:18.599052 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.600738 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.602877 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.603852 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.603947 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:18.611825 (+ 7878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:18.611845 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:18.611864 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:18.611913 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:18.620793 (+ 8880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:18.621867 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:18.621877 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:18.621881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:18.622089 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:18.622567 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:18.622572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:18.622575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:18.623117 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:18.623444 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:18.624036 (+ 592us) server_negotiation.cc:299] Negotiation successful
0925 09:14:18.624323 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11508,"thread_start_us":18890,"threads_started":1}
I20260925 09:14:18.626891 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:18.585449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:45179 (local address 127.3.19.65:35243)
0925 09:14:18.590699 (+ 5250us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:18.590719 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:18.592675 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:18.594386 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:18.594394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:18.596197 (+ 1803us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:18.596793 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:18.596809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.598984 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.598992 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:18.602989 (+ 3997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:18.603018 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.604935 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.604945 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:18.605453 (+ 508us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:18.606367 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:18.606409 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:18.611957 (+ 5548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:18.618150 (+ 6193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:18.618162 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:18.618186 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:18.619021 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:18.619541 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:18.619547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:18.619550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:18.622911 (+ 3361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:18.623805 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:18.623814 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:18.624303 (+ 489us) client_negotiation.cc:769] Sending connection context
0925 09:14:18.624647 (+ 344us) client_negotiation.cc:240] Negotiation successful
0925 09:14:18.625268 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5075,"thread_start_us":1470,"threads_started":1}
I20260925 09:14:18.627584 9669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:18.628428 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:18.576786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:35243 (local address 127.3.19.125:45179)
0925 09:14:18.593586 (+ 16800us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:18.593595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:18.593638 (+ 43us) server_negotiation.cc:407] Connection header received
0925 09:14:18.593715 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:18.593723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:18.593842 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:18.593991 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:18.597065 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.598748 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.603238 (+ 4490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.604291 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.604469 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:18.612371 (+ 7902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:18.612382 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:18.612390 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:18.612450 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:18.617947 (+ 5497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:18.619226 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:18.619232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:18.619236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:18.619397 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:18.623086 (+ 3689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:18.623095 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:18.623100 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:18.623554 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:18.623696 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:18.627662 (+ 3966us) server_negotiation.cc:299] Negotiation successful
0925 09:14:18.627957 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16610,"thread_start_us":17343,"threads_started":1}
I20260925 09:14:18.629833 10321 heartbeater.cc:347] Connected to a master server at 127.3.19.125:45179
I20260925 09:14:18.630203 10321 heartbeater.cc:464] Registering TS with master...
Sep 25 09:14:18 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327658, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:18.631371 10321 heartbeater.cc:511] Master 127.3.19.125:45179 requested a full tablet report, sending...
I20260925 09:14:18.634188 9951 ts_manager.cc:194] Registered new tserver with Master: ff75d2f51e324151a2b0a4e0705a5fcb (127.3.19.65:39165)
I20260925 09:14:18.640754 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:18.576961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:40789 (local address 127.3.19.126:39883)
0925 09:14:18.617126 (+ 40165us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:18.617132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:18.617151 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:14:18.617208 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:18.617221 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:18.617332 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:18.617440 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:18.621862 (+ 4422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.623045 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.625554 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.626553 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.626652 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:18.633474 (+ 6822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:18.633481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:18.633486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:18.633540 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:18.636850 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:18.638065 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:18.638075 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:18.638082 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:18.638420 (+ 338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:18.638960 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:18.638967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:18.638974 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:18.639420 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:18.639553 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:18.640093 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 09:14:18.640347 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40013,"thread_start_us":39896,"threads_started":1}
I20260925 09:14:18.640779 10328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:18.575845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.65:40789)
0925 09:14:18.587723 (+ 11878us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:18.587800 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:18.616559 (+ 28759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:18.617722 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:18.617729 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:18.618343 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:18.621646 (+ 3303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:18.621668 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.623222 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.623228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:18.625384 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:18.625397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.626989 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.626997 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:18.627272 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:18.628076 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:18.628099 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:18.633191 (+ 5092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:18.637011 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:18.637019 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:18.637027 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:18.637894 (+ 867us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:18.638566 (+ 672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:18.638574 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:18.638581 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:18.638825 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:18.639629 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:18.639634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:18.639773 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 09:14:18.639905 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 09:14:18.640261 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11383,"thread_start_us":1659,"threads_started":1}
I20260925 09:14:18.641606 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:18.585524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:39567 (local address 127.3.19.124:38283)
0925 09:14:18.585731 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:18.585740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:18.590948 (+ 5208us) server_negotiation.cc:407] Connection header received
0925 09:14:18.592278 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:18.592289 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:18.592406 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:18.592499 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:18.596600 (+ 4101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.597993 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.614730 (+ 16737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.615583 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.615702 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:18.631653 (+ 15951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:18.631664 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:18.631672 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:18.631731 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:18.636781 (+ 5050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:18.638298 (+ 1517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:18.638308 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:18.638314 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:18.638555 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:18.639079 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:18.639085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:18.639087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:18.639443 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:18.639629 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:18.640365 (+ 736us) server_negotiation.cc:299] Negotiation successful
0925 09:14:18.640616 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 09:14:18.641691 10323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:18.585916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:38283 (local address 127.3.19.65:39567)
0925 09:14:18.590849 (+ 4933us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:18.590869 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:18.591820 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:18.593329 (+ 1509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:18.593347 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:18.595004 (+ 1657us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:18.596133 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:18.596160 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.598238 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.598244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:18.614485 (+ 16241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:18.614507 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:18.616041 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:18.616048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:18.616254 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:18.618283 (+ 2029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:18.618305 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:18.631379 (+ 13074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:18.637048 (+ 5669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:18.637057 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:18.637064 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:18.638098 (+ 1034us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:18.638718 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:18.638726 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:18.638732 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:18.638945 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:18.639577 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:18.639582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:18.639754 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 09:14:18.639943 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 09:14:18.641123 (+ 1180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4804,"mutex_wait_us":844,"spinlock_wait_cycles":17280,"thread_start_us":4415,"threads_started":1}
I20260925 09:14:18.642063 10319 heartbeater.cc:347] Connected to a master server at 127.3.19.126:39883
I20260925 09:14:18.642237 10319 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:18.642827 10319 heartbeater.cc:511] Master 127.3.19.126:39883 requested a full tablet report, sending...
I20260925 09:14:18.643036 10320 heartbeater.cc:347] Connected to a master server at 127.3.19.124:38283
I20260925 09:14:18.643198 10320 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:18.643790 10320 heartbeater.cc:511] Master 127.3.19.124:38283 requested a full tablet report, sending...
I20260925 09:14:18.644356 9351 ts_manager.cc:194] Re-registered known tserver with Master: ff75d2f51e324151a2b0a4e0705a5fcb (127.3.19.65:39165)
I20260925 09:14:18.645625 9351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:40789
I20260925 09:14:18.646690 10141 ts_manager.cc:194] Registered new tserver with Master: ff75d2f51e324151a2b0a4e0705a5fcb (127.3.19.65:39165)
I20260925 09:14:18.648699 10101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: NON_VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: true } } }
I20260925 09:14:18.649099 10101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:18.649988 10331 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a: attempting to promote NON_VOTER 1ffbdef9809940d5bf56c324e26bd133 to VOTER
I20260925 09:14:18.650660 10331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } }
I20260925 09:14:18.652642 10151 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 LEARNER]: Refusing update from remote peer 3cd59f740532408ba0b9dd8b2012096a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 09:14:18.653117 9961 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 FOLLOWER]: Refusing update from remote peer 3cd59f740532408ba0b9dd8b2012096a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 09:14:18.653669 10101 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 09:14:18.654094 10331 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 09:14:18.656772 10101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1ffbdef9809940d5bf56c324e26bd133 (127.3.19.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } } }
I20260925 09:14:18.660369 10101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } } }
I20260925 09:14:18.660676 10101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:18.661512 10331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } } }
I20260925 09:14:18.661861 10331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:18.662089 10331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } } }
I20260925 09:14:18.662344 10331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:18.662196 9961 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1ffbdef9809940d5bf56c324e26bd133 (127.3.19.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } } }
I20260925 09:14:18.663439 10151 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1ffbdef9809940d5bf56c324e26bd133 (127.3.19.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } } }
I20260925 09:14:18.669317 10179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } } }
I20260925 09:14:18.669684 10179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:14:18.672271 10336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3cd59f740532408ba0b9dd8b2012096a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } } }
I20260925 09:14:18.672785 10336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:14:18.734524 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 9546
I20260925 09:14:18.751092 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:36105
--local_ip_for_outbound_sockets=127.3.19.66
--tserver_master_addrs=127.3.19.126:39883
--webserver_port=40533
--webserver_interface=127.3.19.66
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab
--principal=oryx/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.19.126:39883,127.3.19.125:45179
--tserver_master_addrs=127.3.19.126:39883,127.3.19.125:45179,127.3.19.124:38283 with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:19.095657 10337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:19.096066 10337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:19.096211 10337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:19.106457 10337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:19.106581 10337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:19.106700 10337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:14:19.116688 10337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab
--principal=oryx/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:36105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=40533
--tserver_master_addrs=127.3.19.126:39883,127.3.19.125:45179,127.3.19.124:38283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:19.118264 10337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:19.119637 10337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:19.134985 10343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:19.135528 10345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:19.135567 10342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:19.139004 10337 server_base.cc:1034] running on GCE node
I20260925 09:14:19.139782 10337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:19.140810 10337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:19.142073 10337 hybrid_clock.cc:648] HybridClock initialized: now 1790327659141997 us; error 78 us; skew 500 ppm
I20260925 09:14:19.142438 10337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:19 dist-test-slave-l6qh krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327659, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:19.146000 10337 init.cc:377] Logged in from keytab as oryx/127.3.19.66@KRBTEST.COM (short username oryx)
I20260925 09:14:19.147614 10337 webserver.cc:460] Webserver started at http://127.3.19.66:40533/ using document root <none> and password file <none>
I20260925 09:14:19.148396 10337 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:19.148520 10337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:19.154819 10337 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 09:14:19.157836 10352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:19.159528 10337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:19.159684 10337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "80c500977f6c46049c7d88b40d4c582d"
format_stamp: "Formatted at 2026-09-25 09:14:15 on dist-test-slave-l6qh"
server_key: "3b84ebc37cf28021cdccd0f79eb28851"
server_key_iv: "7d1f31dc7b238b1d544d791547811870"
server_key_version: "encryptionkey@0"
I20260925 09:14:19.160785 10337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:19.189483 10337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:19.194587 10337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:19.194928 10337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:19.196017 10337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:19.197597 10337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:19.197727 10337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:19.197867 10337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:19.197964 10337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:19.246075 10337 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:36105
I20260925 09:14:19.246181 10465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:36105 every 8 connection(s)
I20260925 09:14:19.247612 10337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:14:19.257299 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 10337
I20260925 09:14:19.257629 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 9680
Sep 25 09:14:19 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327659, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.125@KRBTEST.COM
Sep 25 09:14:19 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327659, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
Sep 25 09:14:19 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327659, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.124@KRBTEST.COM
Sep 25 09:14:19 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.19.67@KRBTEST.COM
I20260925 09:14:19.279915 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.250834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:39291 (local address 127.3.19.125:45179)
0925 09:14:19.251290 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:19.251296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:19.252670 (+ 1374us) server_negotiation.cc:407] Connection header received
0925 09:14:19.255393 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:19.255400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:19.255563 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:19.255652 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:19.258566 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.259791 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.262132 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.262833 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.262942 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.269559 (+ 6617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:19.269569 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:19.269576 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:19.269620 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:19.274977 (+ 5357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.276273 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.276279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.276282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.276535 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.277193 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.277200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.277204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.277652 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:19.277802 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:19.279117 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0925 09:14:19.279416 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":216,"threads_started":1}
I20260925 09:14:19.281962 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.250812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:33763 (local address 127.3.19.124:38283)
0925 09:14:19.251494 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:19.251499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:19.252962 (+ 1463us) server_negotiation.cc:407] Connection header received
0925 09:14:19.255427 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:19.255434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:19.255524 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:19.255609 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:19.262050 (+ 6441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.263138 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.265192 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.266178 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.266297 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.272884 (+ 6587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:19.272890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:19.272894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:19.272969 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:19.278217 (+ 5248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.279180 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.279186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.279189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.279397 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.279961 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.279969 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.279974 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.280365 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:19.280507 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:19.281087 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 09:14:19.281349 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":458,"threads_started":1}
I20260925 09:14:19.281991 10475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.253326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:41115 (local address 127.3.19.126:39883)
0925 09:14:19.254100 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:19.254106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:19.254127 (+ 21us) server_negotiation.cc:407] Connection header received
0925 09:14:19.255582 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:19.255591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:19.255669 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:19.255765 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:19.259387 (+ 3622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.260838 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.263170 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.263851 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.263987 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.271114 (+ 7127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:19.271120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:19.271124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:19.271162 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:19.277255 (+ 6093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.278124 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.278132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.278136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.278344 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.279565 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.279573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.279577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.280455 (+ 878us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:19.280626 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:19.281150 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 09:14:19.281457 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":520,"threads_started":1}
I20260925 09:14:19.282738 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.253182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.66:41115)
0925 09:14:19.253876 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:19.253897 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:19.255109 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:19.256149 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:19.256153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:19.257194 (+ 1041us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:19.259148 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:19.259170 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.261068 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.261077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:19.262995 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:19.263008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.264234 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.264240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:19.264906 (+ 666us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.265800 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:19.265876 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:19.270828 (+ 4952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:19.277401 (+ 6573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:19.277408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:19.277413 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:19.277956 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:19.279180 (+ 1224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:19.279185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:19.279190 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:19.279428 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:19.280582 (+ 1154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:19.280588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:19.280811 (+ 223us) client_negotiation.cc:769] Sending connection context
0925 09:14:19.281036 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 09:14:19.281372 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":152,"spinlock_wait_cycles":5632,"thread_start_us":306,"threads_started":1}
I20260925 09:14:19.283135 10472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.251518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:38283 (local address 127.3.19.66:33763)
0925 09:14:19.252456 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:19.252513 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:19.255038 (+ 2525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:19.256113 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:19.256126 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:19.257194 (+ 1068us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:19.261837 (+ 4643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:19.261857 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.263338 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.263343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:19.264984 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:19.265002 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.266759 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.266767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:19.267009 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.267898 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:19.267920 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:19.272608 (+ 4688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:19.278382 (+ 5774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:19.278386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:19.278390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:19.279017 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:19.279560 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:19.279565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:19.279569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:19.279823 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:19.280583 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:19.280587 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:19.280807 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 09:14:19.282227 (+ 1420us) client_negotiation.cc:240] Negotiation successful
0925 09:14:19.282548 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":152,"spinlock_wait_cycles":1920,"thread_start_us":406,"threads_started":1}
I20260925 09:14:19.283242 10473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.251746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:45179 (local address 127.3.19.66:39291)
0925 09:14:19.252545 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:19.252566 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:19.255053 (+ 2487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:19.256115 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:19.256127 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:19.257308 (+ 1181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:19.258343 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:19.258377 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.259963 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.259969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:19.261947 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:19.261962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.263356 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.263367 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:19.263758 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.264816 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:19.264855 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:19.269192 (+ 4337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:19.275177 (+ 5985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:19.275189 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:19.275215 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:19.276139 (+ 924us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:19.276697 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:19.276702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:19.276707 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:19.277037 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:19.277939 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:19.277953 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:19.278495 (+ 542us) client_negotiation.cc:769] Sending connection context
0925 09:14:19.280255 (+ 1760us) client_negotiation.cc:240] Negotiation successful
0925 09:14:19.280883 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":49,"spinlock_wait_cycles":348160,"thread_start_us":499,"threads_started":1}
I20260925 09:14:19.284859 10466 heartbeater.cc:347] Connected to a master server at 127.3.19.126:39883
I20260925 09:14:19.284976 10467 heartbeater.cc:347] Connected to a master server at 127.3.19.124:38283
I20260925 09:14:19.285249 10467 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:19.285250 10466 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:19.284806 10468 heartbeater.cc:347] Connected to a master server at 127.3.19.125:45179
I20260925 09:14:19.285856 10468 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:19.286866 10466 heartbeater.cc:511] Master 127.3.19.126:39883 requested a full tablet report, sending...
I20260925 09:14:19.286918 10467 heartbeater.cc:511] Master 127.3.19.124:38283 requested a full tablet report, sending...
I20260925 09:14:19.286958 10468 heartbeater.cc:511] Master 127.3.19.125:45179 requested a full tablet report, sending...
I20260925 09:14:19.288838 9351 ts_manager.cc:194] Re-registered known tserver with Master: 80c500977f6c46049c7d88b40d4c582d (127.3.19.66:36105)
I20260925 09:14:19.288882 10141 ts_manager.cc:194] Registered new tserver with Master: 80c500977f6c46049c7d88b40d4c582d (127.3.19.66:36105)
I20260925 09:14:19.288991 9951 ts_manager.cc:194] Registered new tserver with Master: 80c500977f6c46049c7d88b40d4c582d (127.3.19.66:36105)
I20260925 09:14:19.290138 9351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:41115
I20260925 09:14:19.294632 10477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.258713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.3.19.67:35667)
0925 09:14:19.261853 (+ 3140us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:19.261863 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:19.261895 (+ 32us) server_negotiation.cc:407] Connection header received
0925 09:14:19.261978 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:19.261989 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:19.262371 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:19.262539 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:19.265356 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.267387 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.270070 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.271271 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.271543 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.285086 (+ 13543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:19.285102 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:19.285116 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:19.285166 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:19.291013 (+ 5847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.291855 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.291866 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.291870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.292048 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.292517 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.292530 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.292533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.293094 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:19.293506 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:19.293878 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 09:14:19.294180 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2893,"thread_start_us":2736,"threads_started":1}
I20260925 09:14:19.296037 9803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:19.381049 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 9680
I20260925 09:14:19.397346 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:35667
--local_ip_for_outbound_sockets=127.3.19.67
--tserver_master_addrs=127.3.19.126:39883
--webserver_port=44927
--webserver_interface=127.3.19.67
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab
--principal=oryx/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.19.126:39883,127.3.19.125:45179
--tserver_master_addrs=127.3.19.126:39883,127.3.19.125:45179,127.3.19.124:38283 with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:14:19.648664 10319 heartbeater.cc:503] Master 127.3.19.126:39883 was elected leader, sending a full tablet report...
W20260925 09:14:19.746835 10479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:19.747187 10479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:19.747351 10479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:19.755360 10479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:19.755522 10479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:19.755695 10479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:14:19.765990 10479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab
--principal=oryx/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:35667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=44927
--tserver_master_addrs=127.3.19.126:39883,127.3.19.125:45179,127.3.19.124:38283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:19.767391 10479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:19.768786 10479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:19.782450 10484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:19.782585 10485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:19.782851 10487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:19.783593 10479 server_base.cc:1034] running on GCE node
I20260925 09:14:19.784224 10479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:19.785364 10479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:19.786593 10479 hybrid_clock.cc:648] HybridClock initialized: now 1790327659786547 us; error 46 us; skew 500 ppm
I20260925 09:14:19.786919 10479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:19 dist-test-slave-l6qh krb5kdc[9306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327659, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:19.790134 10479 init.cc:377] Logged in from keytab as oryx/127.3.19.67@KRBTEST.COM (short username oryx)
I20260925 09:14:19.791646 10479 webserver.cc:460] Webserver started at http://127.3.19.67:44927/ using document root <none> and password file <none>
I20260925 09:14:19.792346 10479 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:19.792505 10479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:19.798002 10479 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 09:14:19.800781 10494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:19.802605 10479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 09:14:19.802820 10479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "594d6f796cae43afb1ca62740a564f50"
format_stamp: "Formatted at 2026-09-25 09:14:16 on dist-test-slave-l6qh"
server_key: "3afcfa3e67448a96657af7d397ace61e"
server_key_iv: "fbb96eb4bb0db88a38ddb3b3e5e584da"
server_key_version: "encryptionkey@0"
I20260925 09:14:19.803889 10479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:19.827976 10479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:19.832460 10479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:19.832747 10479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:19.833854 10479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:19.835371 10479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:19.835490 10479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:19.835619 10479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:19.835717 10479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:19.869472 10479 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:35667
I20260925 09:14:19.869552 10606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:35667 every 8 connection(s)
I20260925 09:14:19.871003 10479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:14:19.880218 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 10479
Sep 25 09:14:19 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327659, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.124@KRBTEST.COM
Sep 25 09:14:19 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327659, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.125@KRBTEST.COM
Sep 25 09:14:19 dist-test-slave-l6qh krb5kdc[9306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327659, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:19.924233 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.874881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:45179 (local address 127.3.19.67:50615)
0925 09:14:19.889249 (+ 14368us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:19.889283 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:19.889491 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:19.894482 (+ 4991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:19.894490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:19.895276 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:19.895719 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:19.895733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.899465 (+ 3732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.899472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:19.901647 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:19.901667 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.905465 (+ 3798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.905472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:19.905705 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.906263 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:19.906285 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:19.911549 (+ 5264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:19.917386 (+ 5837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:19.917400 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:19.917426 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:19.918498 (+ 1072us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:19.919728 (+ 1230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:19.919737 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:19.919743 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:19.920090 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:19.920902 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:19.920913 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:19.921441 (+ 528us) client_negotiation.cc:769] Sending connection context
0925 09:14:19.921821 (+ 380us) client_negotiation.cc:240] Negotiation successful
0925 09:14:19.922444 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14098,"mutex_wait_us":41,"spinlock_wait_cycles":88192,"thread_start_us":14512,"threads_started":1}
I20260925 09:14:19.926632 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.874005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:50615 (local address 127.3.19.125:45179)
0925 09:14:19.894007 (+ 20002us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:19.894016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:19.894041 (+ 25us) server_negotiation.cc:407] Connection header received
0925 09:14:19.894115 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:19.894124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:19.894220 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:19.897656 (+ 3436us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:19.897856 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.899278 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.904409 (+ 5131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.905203 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.906095 (+ 892us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.911945 (+ 5850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:19.911956 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:19.911964 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:19.912021 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:19.917123 (+ 5102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.918906 (+ 1783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.918915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.918921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.919191 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.920256 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.920262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.920267 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.920699 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:19.924706 (+ 4007us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:19.924873 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 09:14:19.925146 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19796,"thread_start_us":32726,"threads_started":1}
I20260925 09:14:19.928243 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.874881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:38283 (local address 127.3.19.67:59429)
0925 09:14:19.877437 (+ 2556us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:19.877493 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:19.880528 (+ 3035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:19.892485 (+ 11957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:19.892489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:19.895341 (+ 2852us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:19.895775 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:19.895793 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.900959 (+ 5166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.900967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:19.902998 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:19.903013 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.905180 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.905188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:19.905440 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.906113 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:19.906154 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:19.912414 (+ 6260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:19.917657 (+ 5243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:19.917663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:19.917670 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:19.919720 (+ 2050us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:19.924799 (+ 5079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:19.924806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:19.924811 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:19.925010 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:19.927060 (+ 2050us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:19.927066 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:19.927261 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 09:14:19.927408 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 09:14:19.927752 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2237,"mutex_wait_us":264,"thread_start_us":1935,"threads_started":1}
I20260925 09:14:19.928385 10609 heartbeater.cc:347] Connected to a master server at 127.3.19.125:45179
I20260925 09:14:19.929355 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.875612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:59429 (local address 127.3.19.124:38283)
0925 09:14:19.892120 (+ 16508us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:19.892126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:19.892148 (+ 22us) server_negotiation.cc:407] Connection header received
0925 09:14:19.892208 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:19.892213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:19.892286 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:19.893390 (+ 1104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:19.896025 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.897426 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.903161 (+ 5735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.903908 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.905770 (+ 1862us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.912635 (+ 6865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:19.912644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:19.912651 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:19.912691 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:19.917487 (+ 4796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.919879 (+ 2392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.919886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.919890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.920071 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.925203 (+ 5132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.925210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.925213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.925648 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:19.925788 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:19.927613 (+ 1825us) server_negotiation.cc:299] Negotiation successful
0925 09:14:19.928830 (+ 1217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16382,"thread_start_us":16271,"threads_started":1}
I20260925 09:14:19.929693 10609 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:19.930790 10609 heartbeater.cc:511] Master 127.3.19.125:45179 requested a full tablet report, sending...
I20260925 09:14:19.933318 9951 ts_manager.cc:194] Registered new tserver with Master: 594d6f796cae43afb1ca62740a564f50 (127.3.19.67:35667)
I20260925 09:14:19.934574 10625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.903581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.3.19.124:38283)
0925 09:14:19.907574 (+ 3993us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:19.907582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:19.907604 (+ 22us) server_negotiation.cc:407] Connection header received
0925 09:14:19.907684 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:19.907691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:19.907780 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:19.907912 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:19.911017 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.912578 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.914780 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.915654 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.915794 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.920698 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:19.920705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:19.920709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:19.920747 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:19.929332 (+ 8585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.930911 (+ 1579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.930919 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.930922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.931503 (+ 581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.932091 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.932096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.932099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.932461 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:19.932734 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:19.933886 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0925 09:14:19.934154 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3781,"thread_start_us":3657,"threads_started":1}
I20260925 09:14:19.936074 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.878066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.67:54847)
0925 09:14:19.890117 (+ 12051us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:19.890144 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:19.890309 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:19.892053 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:19.892075 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:19.893479 (+ 1404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:19.894975 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:19.895024 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.897665 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.897673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:19.900689 (+ 3016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:19.900713 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.904200 (+ 3487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.904210 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:19.904599 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.906694 (+ 2095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:19.906721 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:19.918907 (+ 12186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:19.925898 (+ 6991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:19.925903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:19.925907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:19.926492 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:19.928649 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:19.928654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:19.928658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:19.928890 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:19.934853 (+ 5963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:19.934859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:19.935047 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 09:14:19.935250 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 09:14:19.935556 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11884,"thread_start_us":11721,"threads_started":1}
I20260925 09:14:19.936313 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.878233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:54847 (local address 127.3.19.126:39883)
0925 09:14:19.891139 (+ 12906us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:19.891180 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:19.891210 (+ 30us) server_negotiation.cc:407] Connection header received
0925 09:14:19.891288 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:19.891297 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:19.891415 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:19.891624 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:19.895908 (+ 4284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.897441 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.903189 (+ 5748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.903899 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.904066 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.920355 (+ 16289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:19.920363 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:19.920367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:19.920411 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:19.924615 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.926687 (+ 2072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.926693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.926698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.926880 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.929065 (+ 2185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.929071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.929075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.934618 (+ 5543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:19.934773 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:19.935562 (+ 789us) server_negotiation.cc:299] Negotiation successful
0925 09:14:19.935811 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5080,"server-negotiator.queue_time_us":12780,"thread_start_us":12658,"threads_started":1}
I20260925 09:14:19.937340 10608 heartbeater.cc:347] Connected to a master server at 127.3.19.124:38283
I20260925 09:14:19.937513 10608 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:19.938179 10608 heartbeater.cc:511] Master 127.3.19.124:38283 requested a full tablet report, sending...
I20260925 09:14:19.938441 10607 heartbeater.cc:347] Connected to a master server at 127.3.19.126:39883
I20260925 09:14:19.938607 10607 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:19.939291 10607 heartbeater.cc:511] Master 127.3.19.126:39883 requested a full tablet report, sending...
I20260925 09:14:19.939611 10141 ts_manager.cc:194] Registered new tserver with Master: 594d6f796cae43afb1ca62740a564f50 (127.3.19.67:35667)
I20260925 09:14:19.940884 9351 ts_manager.cc:194] Re-registered known tserver with Master: 594d6f796cae43afb1ca62740a564f50 (127.3.19.67:35667)
I20260925 09:14:19.941939 9351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:54847
I20260925 09:14:19.948400 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.895321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.3.19.126:39883)
0925 09:14:19.921090 (+ 25769us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:19.921097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:19.921118 (+ 21us) server_negotiation.cc:407] Connection header received
0925 09:14:19.921178 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:19.921183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:19.921299 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:19.921506 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:19.924581 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.926056 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.929315 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.930056 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.930223 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.935464 (+ 5241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:19.935472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:19.935478 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:19.935520 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:19.942299 (+ 6779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.945370 (+ 3071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.945375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.945378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.945579 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.946061 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.946066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.946068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.946489 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:19.946854 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:19.947328 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 09:14:19.947646 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25605,"thread_start_us":25467,"threads_started":1}
I20260925 09:14:19.950291 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.893259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.3.19.125:45179)
0925 09:14:19.922076 (+ 28817us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:19.922083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:19.922102 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:14:19.922153 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:19.922158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:19.922227 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:19.926290 (+ 4063us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:19.926569 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.931425 (+ 4856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.934274 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:19.935188 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:19.936847 (+ 1659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:19.941165 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:19.941173 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:19.941178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:19.941300 (+ 122us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:19.946159 (+ 4859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.947133 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.947137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.947139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.947310 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:19.947820 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:19.947826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:19.947829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:19.948400 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:19.948747 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:19.949404 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 09:14:19.949672 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28677,"thread_start_us":33454,"threads_started":1}
I20260925 09:14:19.954102 9351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:14:19.963706 9351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:14:20.011624 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.988849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.3.19.67:35667)
0925 09:14:19.990303 (+ 1454us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:19.990318 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:19.996530 (+ 6212us) server_negotiation.cc:407] Connection header received
0925 09:14:19.996711 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:19.996724 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:19.997129 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:19.997409 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:20.000624 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.003968 (+ 3344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.007187 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.009729 (+ 2542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.009882 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.010203 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:20.010862 (+ 659us) server_negotiation.cc:299] Negotiation successful
0925 09:14:20.011092 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":1283,"threads_started":1}
I20260925 09:14:20.013471 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.988250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:35667 (local address 127.0.0.1:49850)
0925 09:14:19.996115 (+ 7865us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:19.996137 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:19.996374 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:19.999225 (+ 2851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:19.999233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:19.999273 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:20.000097 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:20.000113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.004186 (+ 4073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.004194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:20.006950 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:20.006969 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.010360 (+ 3391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.010367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:20.010650 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.010705 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 09:14:20.010806 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 09:14:20.010944 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7627,"thread_start_us":7399,"threads_started":1}
I20260925 09:14:20.018764 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.989709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.3.19.66:36105)
0925 09:14:20.000469 (+ 10760us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:20.000482 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:20.002095 (+ 1613us) server_negotiation.cc:407] Connection header received
0925 09:14:20.002232 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:20.002241 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:20.002574 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:20.002821 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:20.004602 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.006844 (+ 2242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.014872 (+ 8028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.017029 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.017158 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.017441 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:20.018145 (+ 704us) server_negotiation.cc:299] Negotiation successful
0925 09:14:20.018389 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10391,"thread_start_us":10753,"threads_started":1}
I20260925 09:14:20.018782 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.988715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:36105 (local address 127.0.0.1:60882)
0925 09:14:20.001769 (+ 13054us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:20.001789 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:20.001953 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:20.003131 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:20.003138 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:20.003174 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:20.004244 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:20.004261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.007057 (+ 2796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.007064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:20.014612 (+ 7548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:20.014629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.017705 (+ 3076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.017710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:20.017939 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.017963 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 09:14:20.018072 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 09:14:20.018231 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12864,"thread_start_us":12551,"threads_started":1}
I20260925 09:14:20.023921 10400 tablet_service.cc:1467] Processing CreateTablet for tablet d5d3dcfd809447a18a8dd1c976dcdaf0 (DEFAULT_TABLE table=test-table [id=ff5bd715f6574508b24fbfd1ab258cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:20.023921 10542 tablet_service.cc:1467] Processing CreateTablet for tablet d5d3dcfd809447a18a8dd1c976dcdaf0 (DEFAULT_TABLE table=test-table [id=ff5bd715f6574508b24fbfd1ab258cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:20.025895 10542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d3dcfd809447a18a8dd1c976dcdaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:20.026427 10400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d3dcfd809447a18a8dd1c976dcdaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:20.032835 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.996444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:39165 (local address 127.0.0.1:33922)
0925 09:14:20.003967 (+ 7523us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:20.003988 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:20.004139 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:20.019526 (+ 15387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:20.019532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:20.019575 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:20.020730 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:20.020753 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.024151 (+ 3398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.024158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:20.028489 (+ 4331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:20.028510 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.031356 (+ 2846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.031364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:20.031744 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.031785 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 09:14:20.031885 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 09:14:20.032144 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7244,"thread_start_us":7801,"threads_started":1}
I20260925 09:14:20.036041 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:19.996743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.3.19.65:39165)
0925 09:14:20.017832 (+ 21089us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:20.017854 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:20.017901 (+ 47us) server_negotiation.cc:407] Connection header received
0925 09:14:20.018088 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:20.018099 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:20.018498 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:20.018839 (+ 341us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:20.021372 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.023971 (+ 2599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.028757 (+ 4786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.031044 (+ 2287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.033317 (+ 2273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.033682 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:20.033775 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 09:14:20.034035 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20512,"thread_start_us":24180,"threads_started":1}
I20260925 09:14:20.041550 10254 tablet_service.cc:1467] Processing CreateTablet for tablet d5d3dcfd809447a18a8dd1c976dcdaf0 (DEFAULT_TABLE table=test-table [id=ff5bd715f6574508b24fbfd1ab258cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:20.044082 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d3dcfd809447a18a8dd1c976dcdaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:20.049853 10645 tablet_bootstrap.cc:492] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50: Bootstrap starting.
I20260925 09:14:20.056582 10645 tablet_bootstrap.cc:654] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:20.058593 10645 log.cc:824] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:20.062958 10644 tablet_bootstrap.cc:492] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d: Bootstrap starting.
I20260925 09:14:20.065364 10645 tablet_bootstrap.cc:492] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50: No bootstrap required, opened a new log
I20260925 09:14:20.065670 10645 ts_tablet_manager.cc:1397] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.006s
I20260925 09:14:20.070267 10647 tablet_bootstrap.cc:492] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb: Bootstrap starting.
I20260925 09:14:20.074482 10644 tablet_bootstrap.cc:654] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:20.075927 10647 tablet_bootstrap.cc:654] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:20.076844 10644 log.cc:824] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:20.078233 10647 log.cc:824] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:20.081643 10645 raft_consensus.cc:348] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } } peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } }
I20260925 09:14:20.082442 10645 raft_consensus.cc:374] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:20.082610 10645 raft_consensus.cc:729] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594d6f796cae43afb1ca62740a564f50, State: Initialized, Role: FOLLOWER
I20260925 09:14:20.083021 10647 tablet_bootstrap.cc:492] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb: No bootstrap required, opened a new log
I20260925 09:14:20.083442 10647 ts_tablet_manager.cc:1397] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.009s
I20260925 09:14:20.083655 10645 consensus_queue.cc:260] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } } peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } }
I20260925 09:14:20.086503 10607 heartbeater.cc:503] Master 127.3.19.126:39883 was elected leader, sending a full tablet report...
I20260925 09:14:20.087301 10645 ts_tablet_manager.cc:1428] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50: Time spent starting tablet: real 0.021s user 0.010s sys 0.008s
I20260925 09:14:20.099671 10644 tablet_bootstrap.cc:492] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d: No bootstrap required, opened a new log
I20260925 09:14:20.100090 10644 ts_tablet_manager.cc:1397] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.003s
I20260925 09:14:20.103039 10647 raft_consensus.cc:348] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } } peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } }
I20260925 09:14:20.103679 10647 raft_consensus.cc:374] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:20.103907 10647 raft_consensus.cc:729] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff75d2f51e324151a2b0a4e0705a5fcb, State: Initialized, Role: FOLLOWER
I20260925 09:14:20.104753 10647 consensus_queue.cc:260] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } } peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } }
I20260925 09:14:20.107295 10647 ts_tablet_manager.cc:1428] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb: Time spent starting tablet: real 0.024s user 0.017s sys 0.005s
I20260925 09:14:20.114075 10644 raft_consensus.cc:348] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } } peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } }
I20260925 09:14:20.114619 10644 raft_consensus.cc:374] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:20.114777 10644 raft_consensus.cc:729] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c500977f6c46049c7d88b40d4c582d, State: Initialized, Role: FOLLOWER
I20260925 09:14:20.115530 10644 consensus_queue.cc:260] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } } peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } }
I20260925 09:14:20.117331 10466 heartbeater.cc:503] Master 127.3.19.126:39883 was elected leader, sending a full tablet report...
I20260925 09:14:20.118036 10644 ts_tablet_manager.cc:1428] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d: Time spent starting tablet: real 0.018s user 0.014s sys 0.006s
W20260925 09:14:20.133714 10610 tablet.cc:2369] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:14:20.251448 10469 tablet.cc:2369] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:14:20.275259 10652 raft_consensus.cc:482] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:14:20.275759 10652 raft_consensus.cc:504] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } } peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } }
I20260925 09:14:20.278127 10652 leader_election.cc:288] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 594d6f796cae43afb1ca62740a564f50 (127.3.19.67:35667), ff75d2f51e324151a2b0a4e0705a5fcb (127.3.19.65:39165)
I20260925 09:14:20.288424 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:20.278748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:53063 (local address 127.3.19.67:35667)
0925 09:14:20.279026 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:20.279031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:20.279877 (+ 846us) server_negotiation.cc:407] Connection header received
0925 09:14:20.280321 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:20.280331 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:20.280437 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:20.280513 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:20.281875 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.283163 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.286029 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.287122 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.287286 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.287466 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:20.287916 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 09:14:20.288056 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 09:14:20.288542 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:20.280042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:41611 (local address 127.3.19.65:39165)
0925 09:14:20.280494 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:20.280500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:20.281157 (+ 657us) server_negotiation.cc:407] Connection header received
0925 09:14:20.281409 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:20.281414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:20.281482 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:20.281585 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:20.282512 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.283396 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.286111 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.287217 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.287372 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.287573 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:20.288074 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 09:14:20.288209 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 09:14:20.288873 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:20.279835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:39165 (local address 127.3.19.66:41611)
0925 09:14:20.281044 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:20.281059 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:20.281265 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:20.281599 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:20.281603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:20.281631 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:20.282312 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:20.282321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.283545 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.283550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:20.285885 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:20.285898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.287694 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.287700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:20.287888 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.287913 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 09:14:20.288020 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 09:14:20.288158 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":43,"thread_start_us":804,"threads_started":1}
I20260925 09:14:20.289163 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:20.278318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:35667 (local address 127.3.19.66:53063)
0925 09:14:20.279763 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:20.279783 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:20.280087 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:20.280869 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:20.280872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:20.280893 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:20.281712 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:20.281728 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.283352 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.283359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:20.285810 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:20.285824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.287562 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.287570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:20.287738 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.287766 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 09:14:20.287887 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 09:14:20.288037 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1272,"mutex_wait_us":26,"thread_start_us":884,"threads_started":1}
I20260925 09:14:20.290442 10562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d3dcfd809447a18a8dd1c976dcdaf0" candidate_uuid: "80c500977f6c46049c7d88b40d4c582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594d6f796cae43afb1ca62740a564f50" is_pre_election: true
I20260925 09:14:20.290449 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d3dcfd809447a18a8dd1c976dcdaf0" candidate_uuid: "80c500977f6c46049c7d88b40d4c582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" is_pre_election: true
I20260925 09:14:20.291203 10562 raft_consensus.cc:2454] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80c500977f6c46049c7d88b40d4c582d in term 0.
I20260925 09:14:20.291206 10274 raft_consensus.cc:2454] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80c500977f6c46049c7d88b40d4c582d in term 0.
I20260925 09:14:20.292246 10356 leader_election.cc:302] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c500977f6c46049c7d88b40d4c582d, ff75d2f51e324151a2b0a4e0705a5fcb; no voters:
I20260925 09:14:20.292841 10652 raft_consensus.cc:2781] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:14:20.293092 10652 raft_consensus.cc:482] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:14:20.293303 10652 raft_consensus.cc:3037] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:20.296211 10652 raft_consensus.cc:504] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } } peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } }
I20260925 09:14:20.297055 10652 leader_election.cc:288] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [CANDIDATE]: Term 1 election: Requested vote from peers 594d6f796cae43afb1ca62740a564f50 (127.3.19.67:35667), ff75d2f51e324151a2b0a4e0705a5fcb (127.3.19.65:39165)
I20260925 09:14:20.297689 10562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d3dcfd809447a18a8dd1c976dcdaf0" candidate_uuid: "80c500977f6c46049c7d88b40d4c582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594d6f796cae43afb1ca62740a564f50"
I20260925 09:14:20.297760 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d3dcfd809447a18a8dd1c976dcdaf0" candidate_uuid: "80c500977f6c46049c7d88b40d4c582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb"
I20260925 09:14:20.297947 10562 raft_consensus.cc:3037] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:20.298082 10274 raft_consensus.cc:3037] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:20.301306 10562 raft_consensus.cc:2454] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80c500977f6c46049c7d88b40d4c582d in term 1.
I20260925 09:14:20.301347 10274 raft_consensus.cc:2454] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80c500977f6c46049c7d88b40d4c582d in term 1.
I20260925 09:14:20.301867 10356 leader_election.cc:302] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 594d6f796cae43afb1ca62740a564f50, 80c500977f6c46049c7d88b40d4c582d; no voters:
I20260925 09:14:20.302212 10652 raft_consensus.cc:2781] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:20.303063 10652 raft_consensus.cc:686] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [term 1 LEADER]: Becoming Leader. State: Replica: 80c500977f6c46049c7d88b40d4c582d, State: Running, Role: LEADER
I20260925 09:14:20.303795 10652 consensus_queue.cc:237] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } } peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } }
I20260925 09:14:20.310755 9351 catalog_manager.cc:5495] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d reported cstate change: term changed from 0 to 1, leader changed from <none> to 80c500977f6c46049c7d88b40d4c582d (127.3.19.66). New cstate: current_term: 1 leader_uuid: "80c500977f6c46049c7d88b40d4c582d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } health_report { overall_health: UNKNOWN } } }
W20260925 09:14:20.353410 10322 tablet.cc:2369] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:14:20.379727 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:20.373406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.3.19.66:36105)
0925 09:14:20.373643 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:20.373648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:20.373667 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:14:20.373865 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:20.373869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:20.373945 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:20.374082 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:14:20.374615 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.375452 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.377124 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.377788 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.377884 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.378420 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:14:20.379040 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:14:20.379259 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:20.379300 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 09:14:20.379366 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 09:14:20.390708 10562 raft_consensus.cc:1261] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50 [term 1 FOLLOWER]: Refusing update from remote peer 80c500977f6c46049c7d88b40d4c582d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:20.390710 10274 raft_consensus.cc:1261] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb [term 1 FOLLOWER]: Refusing update from remote peer 80c500977f6c46049c7d88b40d4c582d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:20.391937 10655 consensus_queue.cc:1035] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:20.392560 10652 consensus_queue.cc:1035] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [LEADER]: Connected to new peer: Peer: permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:20.411742 10659 mvcc.cc:205] Tried to move back new op lower bound from 7333182096948981760 to 7333182096609296384. Current Snapshot: MvccSnapshot[applied={T|T < 7333182096948981760}]
I20260925 09:14:20.432056 9351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:20.432302 9351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:20.439833 9351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cd59f740532408ba0b9dd8b2012096a: Sending DeleteTablet for 3 replicas of tablet d5d3dcfd809447a18a8dd1c976dcdaf0
I20260925 09:14:20.441377 10400 tablet_service.cc:1514] Processing DeleteTablet for tablet d5d3dcfd809447a18a8dd1c976dcdaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:20 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:60882
I20260925 09:14:20.441699 10542 tablet_service.cc:1514] Processing DeleteTablet for tablet d5d3dcfd809447a18a8dd1c976dcdaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:20 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:49850
I20260925 09:14:20.442374 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet d5d3dcfd809447a18a8dd1c976dcdaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:20 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:33922
I20260925 09:14:20.443399 10667 tablet_replica.cc:330] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d: stopping tablet replica
I20260925 09:14:20.443986 10667 raft_consensus.cc:2229] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:14:20.444907 10667 raft_consensus.cc:2258] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:20.445996 10668 tablet_replica.cc:330] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50: stopping tablet replica
I20260925 09:14:20.446727 10668 raft_consensus.cc:2229] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:20.447520 10668 raft_consensus.cc:2258] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:20.452823 10667 ts_tablet_manager.cc:1905] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:20.455479 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 9319
I20260925 09:14:20.455533 10669 tablet_replica.cc:330] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb: stopping tablet replica
I20260925 09:14:20.456182 10669 raft_consensus.cc:2229] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:20.456910 9381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:20.461216 10669 raft_consensus.cc:2258] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:20.462044 10668 ts_tablet_manager.cc:1905] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:20.463882 10669 ts_tablet_manager.cc:1905] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:20.474231 10669 ts_tablet_manager.cc:1918] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:20.474534 10669 log.cc:1196] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/d5d3dcfd809447a18a8dd1c976dcdaf0
I20260925 09:14:20.475582 10669 ts_tablet_manager.cc:1939] T d5d3dcfd809447a18a8dd1c976dcdaf0 P ff75d2f51e324151a2b0a4e0705a5fcb: Deleting consensus metadata
I20260925 09:14:20.490469 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:20.443329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.3.19.125:45179)
0925 09:14:20.443523 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:20.443530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:20.453785 (+ 10255us) server_negotiation.cc:407] Connection header received
0925 09:14:20.453957 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:20.453962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:20.454086 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:20.454201 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:20.457369 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.458742 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.463138 (+ 4396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.463940 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.464078 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.479097 (+ 15019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:20.479104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:20.479108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:20.479218 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:20.484747 (+ 5529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:20.485990 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:20.485996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:20.485999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:20.486206 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:20.486911 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:20.486916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:20.486920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:20.487461 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:20.487837 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:20.489575 (+ 1738us) server_negotiation.cc:299] Negotiation successful
0925 09:14:20.489889 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 09:14:20.492172 10668 ts_tablet_manager.cc:1918] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:20.492543 10668 log.cc:1196] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/d5d3dcfd809447a18a8dd1c976dcdaf0
I20260925 09:14:20.496589 10667 ts_tablet_manager.cc:1918] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:20.497555 10667 log.cc:1196] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/d5d3dcfd809447a18a8dd1c976dcdaf0
I20260925 09:14:20.498267 10668 ts_tablet_manager.cc:1939] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 594d6f796cae43afb1ca62740a564f50: Deleting consensus metadata
I20260925 09:14:20.498898 10667 ts_tablet_manager.cc:1939] T d5d3dcfd809447a18a8dd1c976dcdaf0 P 80c500977f6c46049c7d88b40d4c582d: Deleting consensus metadata
I20260925 09:14:20.509855 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:20.454045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.3.19.124:38283)
0925 09:14:20.481285 (+ 27240us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:20.481293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:20.489288 (+ 7995us) server_negotiation.cc:407] Connection header received
0925 09:14:20.489462 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:20.489477 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:20.489600 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:20.489719 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:20.493450 (+ 3731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.494274 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.496777 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.497605 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.497796 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.502341 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:20.502349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:20.502354 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:20.502398 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:20.506454 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:20.507347 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:20.507354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:20.507358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:20.507538 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:20.508000 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:20.508004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:20.508006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:20.508403 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:20.508759 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:20.509112 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 09:14:20.509393 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27050,"thread_start_us":27321,"threads_started":1}
I20260925 09:14:20.679666 9338 catalog_manager.cc:4841] TS 594d6f796cae43afb1ca62740a564f50 (127.3.19.67:35667): tablet d5d3dcfd809447a18a8dd1c976dcdaf0 (table test-table [id=ff5bd715f6574508b24fbfd1ab258cbf]) successfully deleted
I20260925 09:14:20.682420 9337 catalog_manager.cc:4841] TS 80c500977f6c46049c7d88b40d4c582d (127.3.19.66:36105): tablet d5d3dcfd809447a18a8dd1c976dcdaf0 (table test-table [id=ff5bd715f6574508b24fbfd1ab258cbf]) successfully deleted
I20260925 09:14:20.682876 9338 catalog_manager.cc:4841] TS ff75d2f51e324151a2b0a4e0705a5fcb (127.3.19.65:39165): tablet d5d3dcfd809447a18a8dd1c976dcdaf0 (table test-table [id=ff5bd715f6574508b24fbfd1ab258cbf]) successfully deleted
I20260925 09:14:20.683487 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 9319
I20260925 09:14:20.739508 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:20.720534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.3.19.125:45179)
0925 09:14:20.720792 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:20.720801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:20.722136 (+ 1335us) server_negotiation.cc:407] Connection header received
0925 09:14:20.722474 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:20.722481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:20.722576 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:20.722658 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:20.725507 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.726613 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.728683 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.729396 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.729522 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.732669 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:20.732675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:20.732681 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:20.732722 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:20.736506 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:20.737378 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:20.737384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:20.737388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:20.737544 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:20.737989 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:20.737993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:20.737995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:20.738273 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:20.738529 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:20.738925 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 09:14:20.739153 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 09:14:20.741883 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:20.721789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55464 (local address 127.3.19.124:38283)
0925 09:14:20.721982 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:20.721988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:20.723368 (+ 1380us) server_negotiation.cc:407] Connection header received
0925 09:14:20.723550 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:20.723557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:20.723635 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:20.723739 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:20.726073 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.727252 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.729137 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:20.729791 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:20.729950 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:20.733944 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:20.733950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:20.733953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:20.733987 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:20.737757 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:20.738600 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:20.738605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:20.738608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:20.738773 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:20.740350 (+ 1577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:20.740355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:20.740358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:20.740772 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:20.741013 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:20.741372 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 09:14:20.741560 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 09:14:21.416288 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.415282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.66:56243)
0925 09:14:21.415945 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:21.416099 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"threads_started":1}
W20260925 09:14:21.417029 10466 heartbeater.cc:644] Failed to heartbeat to 127.3.19.126:39883 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
I20260925 09:14:21.421620 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.420505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.67:44619)
0925 09:14:21.421316 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:21.421449 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":554,"threads_started":1}
W20260925 09:14:21.421973 10607 heartbeater.cc:644] Failed to heartbeat to 127.3.19.126:39883 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
I20260925 09:14:21.422533 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.421730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.65:56039)
0925 09:14:21.422167 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:21.422314 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":248,"threads_started":1}
W20260925 09:14:21.423041 10319 heartbeater.cc:644] Failed to heartbeat to 127.3.19.126:39883 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
I20260925 09:14:21.940831 10682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3cd59f740532408ba0b9dd8b2012096a)
I20260925 09:14:21.941090 10683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3cd59f740532408ba0b9dd8b2012096a)
I20260925 09:14:21.941159 10682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } }
I20260925 09:14:21.941349 10683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } }
I20260925 09:14:21.942644 10682 leader_election.cc:288] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cd59f740532408ba0b9dd8b2012096a (127.3.19.126:39883), 1ffbdef9809940d5bf56c324e26bd133 (127.3.19.124:38283)
I20260925 09:14:21.944005 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.942760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:34786)
0925 09:14:21.943441 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:21.943668 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"threads_started":1}
W20260925 09:14:21.945326 10128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
I20260925 09:14:21.946050 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.942768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:34772)
0925 09:14:21.945563 (+ 2795us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:21.945743 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2620,"thread_start_us":2482,"threads_started":1}
I20260925 09:14:21.946779 10683 leader_election.cc:288] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cd59f740532408ba0b9dd8b2012096a (127.3.19.126:39883), 36fc01d8a2804f92b8ccacfe2302230f (127.3.19.125:45179)
W20260925 09:14:21.947772 9938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
I20260925 09:14:21.950419 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.949813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:34798)
0925 09:14:21.950012 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:21.950188 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 09:14:21.950604 10128 leader_election.cc:334] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3cd59f740532408ba0b9dd8b2012096a (127.3.19.126:39883): Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
I20260925 09:14:21.951627 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.950379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:34800)
0925 09:14:21.951073 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:21.951369 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260925 09:14:21.951785 9938 leader_election.cc:334] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3cd59f740532408ba0b9dd8b2012096a (127.3.19.126:39883): Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
I20260925 09:14:21.955257 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.945872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.3.19.124:38283)
0925 09:14:21.948135 (+ 2263us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:21.948143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:21.948173 (+ 30us) server_negotiation.cc:407] Connection header received
0925 09:14:21.948251 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:21.948259 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:21.948349 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:21.949167 (+ 818us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:21.949473 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:21.950752 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:21.952922 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:21.954227 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:21.954363 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:21.954500 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:21.954768 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 09:14:21.954897 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2070,"thread_start_us":2256,"threads_started":1}
I20260925 09:14:21.955248 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.946148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:38283 (local address 127.0.0.1:55476)
0925 09:14:21.946395 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:21.946417 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:21.946560 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:21.948557 (+ 1997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:21.948566 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:21.948602 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:21.949006 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:21.949017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:21.950940 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:21.950946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:21.952753 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:21.952765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:21.954389 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:21.954393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:21.954599 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:21.954639 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 09:14:21.954739 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 09:14:21.954882 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":6,"thread_start_us":1314,"threads_started":1}
I20260925 09:14:21.955369 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.942777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:45179 (local address 127.0.0.1:48412)
0925 09:14:21.944381 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:21.944406 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:21.944587 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:21.947801 (+ 3214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:21.947806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:21.947852 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:21.948140 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:21.948158 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:21.951316 (+ 3158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:21.951324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:21.952962 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:21.952978 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:21.954576 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:21.954583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:21.954766 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:21.954798 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 09:14:21.954880 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 09:14:21.955027 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1482,"mutex_wait_us":1,"thread_start_us":422,"threads_started":1}
I20260925 09:14:21.955502 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.943038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.3.19.125:45179)
0925 09:14:21.947457 (+ 4419us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:21.947463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:21.947481 (+ 18us) server_negotiation.cc:407] Connection header received
0925 09:14:21.947554 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:21.947558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:21.947635 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:21.948365 (+ 730us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:21.948742 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:21.951168 (+ 2426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:21.953125 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:21.954372 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:21.954489 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:21.954594 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:21.954917 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 09:14:21.955070 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":676,"server-negotiator.queue_time_us":4262,"thread_start_us":4089,"threads_started":1}
I20260925 09:14:21.956080 9961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ffbdef9809940d5bf56c324e26bd133" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "36fc01d8a2804f92b8ccacfe2302230f" is_pre_election: true
I20260925 09:14:21.956243 10151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36fc01d8a2804f92b8ccacfe2302230f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1ffbdef9809940d5bf56c324e26bd133" is_pre_election: true
I20260925 09:14:21.956400 9961 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ffbdef9809940d5bf56c324e26bd133 in term 1.
I20260925 09:14:21.956568 10151 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fc01d8a2804f92b8ccacfe2302230f in term 1.
I20260925 09:14:21.956903 10126 leader_election.cc:302] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ffbdef9809940d5bf56c324e26bd133, 36fc01d8a2804f92b8ccacfe2302230f; no voters: 3cd59f740532408ba0b9dd8b2012096a
I20260925 09:14:21.957295 10683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 09:14:21.957417 10683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3cd59f740532408ba0b9dd8b2012096a)
I20260925 09:14:21.957528 10683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 1 FOLLOWER]: Advancing to term 2
I20260925 09:14:21.957702 9938 leader_election.cc:302] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ffbdef9809940d5bf56c324e26bd133, 36fc01d8a2804f92b8ccacfe2302230f; no voters: 3cd59f740532408ba0b9dd8b2012096a
I20260925 09:14:21.958020 10682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 09:14:21.958166 10682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3cd59f740532408ba0b9dd8b2012096a)
I20260925 09:14:21.958310 10682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 1 FOLLOWER]: Advancing to term 2
I20260925 09:14:21.959885 10683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } }
I20260925 09:14:21.960508 10683 leader_election.cc:288] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [CANDIDATE]: Term 2 election: Requested vote from peers 3cd59f740532408ba0b9dd8b2012096a (127.3.19.126:39883), 36fc01d8a2804f92b8ccacfe2302230f (127.3.19.125:45179)
I20260925 09:14:21.960446 10682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } }
I20260925 09:14:21.961092 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.960691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:34810)
0925 09:14:21.960847 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:21.960926 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 09:14:21.961171 10682 leader_election.cc:288] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [CANDIDATE]: Term 2 election: Requested vote from peers 3cd59f740532408ba0b9dd8b2012096a (127.3.19.126:39883), 1ffbdef9809940d5bf56c324e26bd133 (127.3.19.124:38283)
I20260925 09:14:21.961169 9961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ffbdef9809940d5bf56c324e26bd133" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "36fc01d8a2804f92b8ccacfe2302230f"
I20260925 09:14:21.961699 9961 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ffbdef9809940d5bf56c324e26bd133 in current term 2: Already voted for candidate 36fc01d8a2804f92b8ccacfe2302230f in this term.
I20260925 09:14:21.961740 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.961318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:34816)
0925 09:14:21.961490 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:21.961573 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 09:14:21.961894 10151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36fc01d8a2804f92b8ccacfe2302230f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1ffbdef9809940d5bf56c324e26bd133"
I20260925 09:14:21.962179 10151 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 36fc01d8a2804f92b8ccacfe2302230f in current term 2: Already voted for candidate 1ffbdef9809940d5bf56c324e26bd133 in this term.
I20260925 09:14:21.962781 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.962324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:34818)
0925 09:14:21.962561 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:21.962655 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 09:14:21.962805 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:21.962381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:34826)
0925 09:14:21.962568 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:21.962688 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260925 09:14:21.962970 10128 leader_election.cc:334] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3cd59f740532408ba0b9dd8b2012096a (127.3.19.126:39883): Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
I20260925 09:14:21.963116 10128 leader_election.cc:302] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ffbdef9809940d5bf56c324e26bd133; no voters: 36fc01d8a2804f92b8ccacfe2302230f, 3cd59f740532408ba0b9dd8b2012096a
I20260925 09:14:21.963387 10683 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260925 09:14:21.963598 9938 leader_election.cc:334] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3cd59f740532408ba0b9dd8b2012096a (127.3.19.126:39883): Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
I20260925 09:14:21.963706 9938 leader_election.cc:302] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36fc01d8a2804f92b8ccacfe2302230f; no voters: 1ffbdef9809940d5bf56c324e26bd133, 3cd59f740532408ba0b9dd8b2012096a
I20260925 09:14:21.963910 10682 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 09:14:22.418673 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:22.417756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.66:35447)
0925 09:14:22.418301 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:22.418427 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":325,"threads_started":1}
I20260925 09:14:22.423491 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:22.422723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.67:35255)
0925 09:14:22.423234 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:22.423332 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":292,"threads_started":1}
I20260925 09:14:22.424857 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:22.423754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.65:45331)
0925 09:14:22.424518 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:22.424620 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":528,"threads_started":1}
I20260925 09:14:23.420538 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.419744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.66:34843)
0925 09:14:23.420210 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.420334 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":245,"threads_started":1}
I20260925 09:14:23.425220 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.424377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.67:41787)
0925 09:14:23.424888 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.425007 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":297,"threads_started":1}
I20260925 09:14:23.426819 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.425805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.65:42947)
0925 09:14:23.426452 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.426582 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":442,"threads_started":1}
I20260925 09:14:23.792100 10698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:14:23.792366 10698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } }
I20260925 09:14:23.793099 10698 leader_election.cc:288] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3cd59f740532408ba0b9dd8b2012096a (127.3.19.126:39883), 36fc01d8a2804f92b8ccacfe2302230f (127.3.19.125:45179)
I20260925 09:14:23.793690 9961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ffbdef9809940d5bf56c324e26bd133" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "36fc01d8a2804f92b8ccacfe2302230f" is_pre_election: true
I20260925 09:14:23.793944 9961 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ffbdef9809940d5bf56c324e26bd133 in term 2.
I20260925 09:14:23.794241 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.793320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:35124)
0925 09:14:23.793906 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.794024 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":328,"threads_started":1}
I20260925 09:14:23.794409 10126 leader_election.cc:302] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ffbdef9809940d5bf56c324e26bd133, 36fc01d8a2804f92b8ccacfe2302230f; no voters:
I20260925 09:14:23.794771 10698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 09:14:23.794914 10698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:14:23.795032 10698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 2 FOLLOWER]: Advancing to term 3
I20260925 09:14:23.795778 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.795380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:35138)
0925 09:14:23.795549 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.795631 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 09:14:23.795979 10128 leader_election.cc:334] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3cd59f740532408ba0b9dd8b2012096a (127.3.19.126:39883): Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
I20260925 09:14:23.797449 10698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } }
I20260925 09:14:23.797996 10698 leader_election.cc:288] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [CANDIDATE]: Term 3 election: Requested vote from peers 3cd59f740532408ba0b9dd8b2012096a (127.3.19.126:39883), 36fc01d8a2804f92b8ccacfe2302230f (127.3.19.125:45179)
I20260925 09:14:23.798475 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.798066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:35140)
0925 09:14:23.798232 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.798320 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 09:14:23.798508 9961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ffbdef9809940d5bf56c324e26bd133" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "36fc01d8a2804f92b8ccacfe2302230f"
I20260925 09:14:23.798780 9961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 2 FOLLOWER]: Advancing to term 3
I20260925 09:14:23.799324 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.799023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:35148)
0925 09:14:23.799177 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.799234 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 09:14:23.799472 10128 leader_election.cc:334] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3cd59f740532408ba0b9dd8b2012096a (127.3.19.126:39883): Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
I20260925 09:14:23.800915 9961 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ffbdef9809940d5bf56c324e26bd133 in term 3.
I20260925 09:14:23.801374 10126 leader_election.cc:302] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ffbdef9809940d5bf56c324e26bd133, 36fc01d8a2804f92b8ccacfe2302230f; no voters: 3cd59f740532408ba0b9dd8b2012096a
I20260925 09:14:23.801671 10698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 09:14:23.802062 10698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [term 3 LEADER]: Becoming Leader. State: Replica: 1ffbdef9809940d5bf56c324e26bd133, State: Running, Role: LEADER
I20260925 09:14:23.802428 10698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } }
I20260925 09:14:23.804435 10701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ffbdef9809940d5bf56c324e26bd133. Latest consensus state: current_term: 3 leader_uuid: "1ffbdef9809940d5bf56c324e26bd133" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } } }
I20260925 09:14:23.804816 10701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:23.805560 10703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:14:23.809607 10703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:14:23.810469 10703 catalog_manager.cc:1259] Loaded cluster ID: c0376b7b833547c7924a8ec1c3637e51
I20260925 09:14:23.810572 10703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:14:23.811795 10703 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:14:23.812619 10703 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: Loaded TSK: 0
I20260925 09:14:23.813578 10703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:14:23.865060 10141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:14:23.872370 10141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:14:23.876739 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.876206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:35164)
0925 09:14:23.876397 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.876562 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 09:14:23.876850 9961 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [term 3 FOLLOWER]: Refusing update from remote peer 1ffbdef9809940d5bf56c324e26bd133: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 09:14:23.877574 10701 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 09:14:23.877851 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.877434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:35170)
0925 09:14:23.877602 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.877670 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 09:14:23.878044 10128 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 -> Peer 3cd59f740532408ba0b9dd8b2012096a (127.3.19.126:39883): Couldn't send request to peer 3cd59f740532408ba0b9dd8b2012096a. Status: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 09:14:23.880890 10705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ffbdef9809940d5bf56c324e26bd133. Latest consensus state: current_term: 3 leader_uuid: "1ffbdef9809940d5bf56c324e26bd133" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } } }
I20260925 09:14:23.881296 10705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:14:23.882355 10698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "1ffbdef9809940d5bf56c324e26bd133" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } } }
I20260925 09:14:23.882684 10698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:23.885440 10706 mvcc.cc:205] Tried to move back new op lower bound from 7333182111232278528 to 7333182110938333184. Current Snapshot: MvccSnapshot[applied={T|T < 7333182111232278528}]
I20260925 09:14:23.886116 10705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "1ffbdef9809940d5bf56c324e26bd133" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3cd59f740532408ba0b9dd8b2012096a" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 39883 } } peers { permanent_uuid: "36fc01d8a2804f92b8ccacfe2302230f" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 45179 } attrs { promote: false } } peers { permanent_uuid: "1ffbdef9809940d5bf56c324e26bd133" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 38283 } attrs { promote: false } } }
I20260925 09:14:23.886435 10705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fc01d8a2804f92b8ccacfe2302230f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:14:23.899713 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.891475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:36105 (local address 127.0.0.1:60904)
0925 09:14:23.892099 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.892124 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:23.892270 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:23.893875 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:23.893878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:23.893897 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:23.894119 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:23.894127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:23.895756 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:23.895760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:23.897059 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:23.897073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:23.898853 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:23.898859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:23.899070 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:23.899099 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 09:14:23.899237 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 09:14:23.899395 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480}
I20260925 09:14:23.899773 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.891743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.3.19.66:36105)
0925 09:14:23.893491 (+ 1748us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:23.893501 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:23.893535 (+ 34us) server_negotiation.cc:407] Connection header received
0925 09:14:23.893623 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:23.893631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:23.893726 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:23.893851 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:23.894272 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:23.895607 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:23.897261 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:23.898638 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:23.898798 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:23.898931 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:23.899260 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 09:14:23.899395 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1586,"thread_start_us":1383,"threads_started":1}
I20260925 09:14:23.900588 10400 tablet_service.cc:1467] Processing CreateTablet for tablet 7afb908a9ed348fc878f508d2d1f3b49 (DEFAULT_TABLE table=test-table [id=71caf7538f6d42738f58e20cffc81058]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:23.901104 10400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afb908a9ed348fc878f508d2d1f3b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:23.905309 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.891883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:39165 (local address 127.0.0.1:33926)
0925 09:14:23.892513 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.892530 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:23.892651 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:23.898873 (+ 6222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:23.898877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:23.898905 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:23.899366 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:23.899378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:23.900922 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:23.900927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:23.903096 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:23.903107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:23.904385 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:23.904391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:23.904582 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:23.904618 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 09:14:23.904706 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 09:14:23.904849 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":535,"threads_started":1}
I20260925 09:14:23.906103 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.892260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.3.19.65:39165)
0925 09:14:23.898483 (+ 6223us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:23.898489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:23.898517 (+ 28us) server_negotiation.cc:407] Connection header received
0925 09:14:23.898589 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:23.898594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:23.898699 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:23.899539 (+ 840us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:23.899735 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:23.900771 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:23.903276 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:23.904175 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:23.904333 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:23.904448 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:23.905663 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0925 09:14:23.905801 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6094,"thread_start_us":5977,"threads_started":1}
I20260925 09:14:23.906147 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.892737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:35667 (local address 127.0.0.1:49856)
0925 09:14:23.893314 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.893329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:23.893419 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:23.898443 (+ 5024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:23.898448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:23.898476 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:23.898829 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:23.898844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:23.900540 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:23.900545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:23.902851 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:23.902865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:23.905158 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:23.905164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:23.905368 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:23.905558 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 09:14:23.905651 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 09:14:23.905804 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":2126,"threads_started":1}
I20260925 09:14:23.906982 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.893119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.3.19.67:35667)
0925 09:14:23.897690 (+ 4571us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:23.897699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:23.897722 (+ 23us) server_negotiation.cc:407] Connection header received
0925 09:14:23.897791 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:23.897797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:23.897867 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:23.897963 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:23.899038 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:23.900354 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:23.903044 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:23.904769 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:23.904893 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:23.904987 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:23.906535 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0925 09:14:23.906681 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4432,"thread_start_us":4292,"threads_started":1}
I20260925 09:14:23.906899 10254 tablet_service.cc:1467] Processing CreateTablet for tablet 7afb908a9ed348fc878f508d2d1f3b49 (DEFAULT_TABLE table=test-table [id=71caf7538f6d42738f58e20cffc81058]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:23.907384 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afb908a9ed348fc878f508d2d1f3b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:23.907869 10716 tablet_bootstrap.cc:492] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d: Bootstrap starting.
I20260925 09:14:23.907816 10542 tablet_service.cc:1467] Processing CreateTablet for tablet 7afb908a9ed348fc878f508d2d1f3b49 (DEFAULT_TABLE table=test-table [id=71caf7538f6d42738f58e20cffc81058]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:23.908257 10542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afb908a9ed348fc878f508d2d1f3b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:23.909744 10716 tablet_bootstrap.cc:654] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:23.911654 10716 tablet_bootstrap.cc:492] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d: No bootstrap required, opened a new log
I20260925 09:14:23.911759 10716 ts_tablet_manager.cc:1397] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 09:14:23.912359 10716 raft_consensus.cc:348] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } } peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } }
I20260925 09:14:23.912528 10716 raft_consensus.cc:374] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:23.912586 10716 raft_consensus.cc:729] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c500977f6c46049c7d88b40d4c582d, State: Initialized, Role: FOLLOWER
I20260925 09:14:23.912822 10716 consensus_queue.cc:260] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } } peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } }
I20260925 09:14:23.913687 10716 ts_tablet_manager.cc:1428] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 09:14:23.914584 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.914184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.66:38091)
0925 09:14:23.914338 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.914416 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 09:14:23.915266 10719 tablet_bootstrap.cc:492] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50: Bootstrap starting.
I20260925 09:14:23.915884 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.915405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.66:44705)
0925 09:14:23.915637 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.915735 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260925 09:14:23.918118 10467 heartbeater.cc:503] Master 127.3.19.124:38283 was elected leader, sending a full tablet report...
I20260925 09:14:23.920153 10719 tablet_bootstrap.cc:654] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:23.921833 10720 tablet_bootstrap.cc:492] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb: Bootstrap starting.
I20260925 09:14:23.922740 10719 tablet_bootstrap.cc:492] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50: No bootstrap required, opened a new log
I20260925 09:14:23.922909 10719 ts_tablet_manager.cc:1397] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 09:14:23.923662 10719 raft_consensus.cc:348] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } } peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } }
I20260925 09:14:23.923843 10720 tablet_bootstrap.cc:654] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:23.923909 10719 raft_consensus.cc:374] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:23.924016 10719 raft_consensus.cc:729] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594d6f796cae43afb1ca62740a564f50, State: Initialized, Role: FOLLOWER
I20260925 09:14:23.924305 10719 consensus_queue.cc:260] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } } peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } }
I20260925 09:14:23.925168 10719 ts_tablet_manager.cc:1428] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 09:14:23.928591 10608 heartbeater.cc:503] Master 127.3.19.124:38283 was elected leader, sending a full tablet report...
I20260925 09:14:23.930038 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.925563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.67:57435)
0925 09:14:23.929196 (+ 3633us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.929368 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3463,"thread_start_us":887,"threads_started":1}
I20260925 09:14:23.931663 10720 tablet_bootstrap.cc:492] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb: No bootstrap required, opened a new log
I20260925 09:14:23.931686 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.931344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.67:46083)
0925 09:14:23.931492 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.931568 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 09:14:23.931831 10720 ts_tablet_manager.cc:1397] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260925 09:14:23.932525 10720 raft_consensus.cc:348] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } } peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } }
I20260925 09:14:23.932715 10720 raft_consensus.cc:374] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:23.932808 10720 raft_consensus.cc:729] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff75d2f51e324151a2b0a4e0705a5fcb, State: Initialized, Role: FOLLOWER
I20260925 09:14:23.933048 10720 consensus_queue.cc:260] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } } peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } }
I20260925 09:14:23.934018 10720 ts_tablet_manager.cc:1428] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 09:14:23.935478 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.934601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.65:59303)
0925 09:14:23.935080 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.935238 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":241,"threads_started":1}
I20260925 09:14:23.936669 10320 heartbeater.cc:503] Master 127.3.19.124:38283 was elected leader, sending a full tablet report...
I20260925 09:14:23.937408 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:23.936870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.65:32943)
0925 09:14:23.937136 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:23.937221 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260925 09:14:24.064383 10718 raft_consensus.cc:482] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:14:24.064695 10718 raft_consensus.cc:504] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } } peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } }
I20260925 09:14:24.065536 10718 leader_election.cc:288] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff75d2f51e324151a2b0a4e0705a5fcb (127.3.19.65:39165), 594d6f796cae43afb1ca62740a564f50 (127.3.19.67:35667)
I20260925 09:14:24.066226 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afb908a9ed348fc878f508d2d1f3b49" candidate_uuid: "80c500977f6c46049c7d88b40d4c582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" is_pre_election: true
I20260925 09:14:24.066432 10274 raft_consensus.cc:2454] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80c500977f6c46049c7d88b40d4c582d in term 0.
I20260925 09:14:24.066291 10562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afb908a9ed348fc878f508d2d1f3b49" candidate_uuid: "80c500977f6c46049c7d88b40d4c582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594d6f796cae43afb1ca62740a564f50" is_pre_election: true
I20260925 09:14:24.066730 10562 raft_consensus.cc:2454] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80c500977f6c46049c7d88b40d4c582d in term 0.
I20260925 09:14:24.066984 10356 leader_election.cc:302] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c500977f6c46049c7d88b40d4c582d, ff75d2f51e324151a2b0a4e0705a5fcb; no voters:
I20260925 09:14:24.067353 10718 raft_consensus.cc:2781] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:14:24.067548 10718 raft_consensus.cc:482] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:14:24.067667 10718 raft_consensus.cc:3037] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:24.069962 10718 raft_consensus.cc:504] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } } peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } }
I20260925 09:14:24.070552 10718 leader_election.cc:288] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [CANDIDATE]: Term 1 election: Requested vote from peers ff75d2f51e324151a2b0a4e0705a5fcb (127.3.19.65:39165), 594d6f796cae43afb1ca62740a564f50 (127.3.19.67:35667)
I20260925 09:14:24.071249 10562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afb908a9ed348fc878f508d2d1f3b49" candidate_uuid: "80c500977f6c46049c7d88b40d4c582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594d6f796cae43afb1ca62740a564f50"
I20260925 09:14:24.071236 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afb908a9ed348fc878f508d2d1f3b49" candidate_uuid: "80c500977f6c46049c7d88b40d4c582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb"
I20260925 09:14:24.071475 10274 raft_consensus.cc:3037] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:24.071467 10562 raft_consensus.cc:3037] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:24.074539 10562 raft_consensus.cc:2454] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80c500977f6c46049c7d88b40d4c582d in term 1.
I20260925 09:14:24.074529 10274 raft_consensus.cc:2454] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80c500977f6c46049c7d88b40d4c582d in term 1.
I20260925 09:14:24.075018 10356 leader_election.cc:302] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c500977f6c46049c7d88b40d4c582d, ff75d2f51e324151a2b0a4e0705a5fcb; no voters:
I20260925 09:14:24.075348 10718 raft_consensus.cc:2781] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:24.075832 10718 raft_consensus.cc:686] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [term 1 LEADER]: Becoming Leader. State: Replica: 80c500977f6c46049c7d88b40d4c582d, State: Running, Role: LEADER
I20260925 09:14:24.076144 10718 consensus_queue.cc:237] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } } peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } }
I20260925 09:14:24.076915 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:24.076475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.66:60099)
0925 09:14:24.076657 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:24.076757 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 09:14:24.079236 10140 catalog_manager.cc:5495] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d reported cstate change: term changed from 0 to 1, leader changed from <none> to 80c500977f6c46049c7d88b40d4c582d (127.3.19.66). New cstate: current_term: 1 leader_uuid: "80c500977f6c46049c7d88b40d4c582d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c500977f6c46049c7d88b40d4c582d" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 36105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 } health_report { overall_health: UNKNOWN } } }
I20260925 09:14:24.110031 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:24.105211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60918 (local address 127.3.19.66:36105)
0925 09:14:24.105456 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:24.105461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:24.105483 (+ 22us) server_negotiation.cc:407] Connection header received
0925 09:14:24.105697 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:24.105702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:24.105766 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:24.105863 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:14:24.106229 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:24.106967 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:24.108318 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:24.108732 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:24.108878 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:24.109299 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:14:24.109412 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:14:24.109523 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:24.109659 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 09:14:24.109748 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 09:14:24.113082 10562 raft_consensus.cc:1261] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50 [term 1 FOLLOWER]: Refusing update from remote peer 80c500977f6c46049c7d88b40d4c582d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:24.113178 10274 raft_consensus.cc:1261] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb [term 1 FOLLOWER]: Refusing update from remote peer 80c500977f6c46049c7d88b40d4c582d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:24.113765 10727 consensus_queue.cc:1035] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [LEADER]: Connected to new peer: Peer: permanent_uuid: "594d6f796cae43afb1ca62740a564f50" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 35667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:24.114163 10718 consensus_queue.cc:1035] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff75d2f51e324151a2b0a4e0705a5fcb" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:24.116456 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:24.115864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.65:51683)
0925 09:14:24.116060 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:24.116192 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 09:14:24.116614 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:24.116070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.67:60497)
0925 09:14:24.116291 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:24.116385 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260925 09:14:24.121780 10731 mvcc.cc:205] Tried to move back new op lower bound from 7333182112201416704 to 7333182112058355712. Current Snapshot: MvccSnapshot[applied={T|T < 7333182112201416704}]
I20260925 09:14:24.122066 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:24.121556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.66:33279)
0925 09:14:24.121764 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:24.121869 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 09:14:24.122771 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:24.122231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.65:33175)
0925 09:14:24.122546 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:24.122631 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260925 09:14:24.132934 10141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:24.133601 10141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:24.134158 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:24.133662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.66:50247)
0925 09:14:24.133850 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:24.133950 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 09:14:24.136754 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:24.136152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.3.19.67:50313)
0925 09:14:24.136439 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:24.136542 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260925 09:14:24.143630 10141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ffbdef9809940d5bf56c324e26bd133: Sending DeleteTablet for 3 replicas of tablet 7afb908a9ed348fc878f508d2d1f3b49
I20260925 09:14:24.144861 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet 7afb908a9ed348fc878f508d2d1f3b49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:24 UTC) from {username='oryx', principal='oryx/127.3.19.124@KRBTEST.COM'} at 127.0.0.1:33926
I20260925 09:14:24.145320 10541 tablet_service.cc:1514] Processing DeleteTablet for tablet 7afb908a9ed348fc878f508d2d1f3b49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:24 UTC) from {username='oryx', principal='oryx/127.3.19.124@KRBTEST.COM'} at 127.0.0.1:49856
I20260925 09:14:24.145429 10400 tablet_service.cc:1514] Processing DeleteTablet for tablet 7afb908a9ed348fc878f508d2d1f3b49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:24 UTC) from {username='oryx', principal='oryx/127.3.19.124@KRBTEST.COM'} at 127.0.0.1:60904
I20260925 09:14:24.145334 10542 tablet_service.cc:1514] Processing DeleteTablet for tablet 7afb908a9ed348fc878f508d2d1f3b49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:24 UTC) from {username='oryx', principal='oryx/127.3.19.124@KRBTEST.COM'} at 127.0.0.1:49856
I20260925 09:14:24.146911 10741 tablet_replica.cc:330] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb: stopping tablet replica
I20260925 09:14:24.148283 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 10124
I20260925 09:14:24.153548 10739 tablet_replica.cc:330] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d: stopping tablet replica
I20260925 09:14:24.151823 10741 raft_consensus.cc:2229] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:24.155262 10741 raft_consensus.cc:2258] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:24.155560 10739 raft_consensus.cc:2229] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:14:24.157119 10740 tablet_replica.cc:330] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50: stopping tablet replica
I20260925 09:14:24.157364 10739 raft_consensus.cc:2258] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:24.157709 10741 ts_tablet_manager.cc:1905] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:24.157737 10740 raft_consensus.cc:2229] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:24.158008 10740 raft_consensus.cc:2258] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:24.158615 10739 ts_tablet_manager.cc:1905] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:24.160355 10740 ts_tablet_manager.cc:1905] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:24.164691 10741 ts_tablet_manager.cc:1918] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:24.164829 10741 log.cc:1196] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/7afb908a9ed348fc878f508d2d1f3b49
I20260925 09:14:24.165205 10739 ts_tablet_manager.cc:1918] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:24.165304 10741 ts_tablet_manager.cc:1939] T 7afb908a9ed348fc878f508d2d1f3b49 P ff75d2f51e324151a2b0a4e0705a5fcb: Deleting consensus metadata
I20260925 09:14:24.165367 10739 log.cc:1196] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/7afb908a9ed348fc878f508d2d1f3b49
I20260925 09:14:24.165818 10739 ts_tablet_manager.cc:1939] T 7afb908a9ed348fc878f508d2d1f3b49 P 80c500977f6c46049c7d88b40d4c582d: Deleting consensus metadata
I20260925 09:14:24.166465 10128 catalog_manager.cc:4841] TS ff75d2f51e324151a2b0a4e0705a5fcb (127.3.19.65:39165): tablet 7afb908a9ed348fc878f508d2d1f3b49 (table test-table [id=71caf7538f6d42738f58e20cffc81058]) successfully deleted
I20260925 09:14:24.167304 10127 catalog_manager.cc:4841] TS 80c500977f6c46049c7d88b40d4c582d (127.3.19.66:36105): tablet 7afb908a9ed348fc878f508d2d1f3b49 (table test-table [id=71caf7538f6d42738f58e20cffc81058]) successfully deleted
I20260925 09:14:24.168208 10740 ts_tablet_manager.cc:1918] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:24.168380 10740 log.cc:1196] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/7afb908a9ed348fc878f508d2d1f3b49
I20260925 09:14:24.168743 10740 ts_tablet_manager.cc:1939] T 7afb908a9ed348fc878f508d2d1f3b49 P 594d6f796cae43afb1ca62740a564f50: Deleting consensus metadata
I20260925 09:14:24.169795 10128 catalog_manager.cc:4841] TS 594d6f796cae43afb1ca62740a564f50 (127.3.19.67:35667): tablet 7afb908a9ed348fc878f508d2d1f3b49 (table test-table [id=71caf7538f6d42738f58e20cffc81058]) successfully deleted
W20260925 09:14:24.170493 10128 catalog_manager.cc:4808] TS 594d6f796cae43afb1ca62740a564f50 (127.3.19.67:35667): delete failed for tablet 7afb908a9ed348fc878f508d2d1f3b49 because the tablet was not found. No further retry: Not found: Tablet not found: 7afb908a9ed348fc878f508d2d1f3b49
I20260925 09:14:24.172952 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:24.149527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.3.19.65:39165)
0925 09:14:24.149756 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:24.149762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:24.152503 (+ 2741us) server_negotiation.cc:407] Connection header received
0925 09:14:24.152575 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:24.152580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:24.152682 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:24.152809 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:24.158159 (+ 5350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:24.159273 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:24.161405 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:24.162137 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:24.162267 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:24.166300 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:24.166318 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:24.166339 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:24.166380 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:24.169869 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:24.170796 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:24.170804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:24.170807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:24.170948 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:24.171356 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:24.171360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:24.171362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:24.171736 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:24.172034 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:24.172357 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 09:14:24.172627 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 09:14:24.173702 10314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:24.270817 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 10124
I20260925 09:14:24.289763 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 10337
I20260925 09:14:24.304113 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:24.290584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.3.19.66:36105)
0925 09:14:24.290795 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:24.290802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:24.290910 (+ 108us) server_negotiation.cc:407] Connection header received
0925 09:14:24.291134 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:24.291138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:24.291210 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:24.291319 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:24.292615 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:24.293443 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:24.294656 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:24.295078 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:24.295211 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:24.297847 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:24.297861 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:24.297878 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:24.297918 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:24.300868 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:24.301772 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:24.301782 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:24.301785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:24.301960 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:24.302371 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:24.302378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:24.302382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:24.302822 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:24.303141 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:24.303468 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 09:14:24.303698 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 09:14:24.304991 10461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:24.318291 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:24.317705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:35174)
0925 09:14:24.317972 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:24.318092 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 09:14:24.319396 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:24.318942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:39883 (local address 127.0.0.1:35180)
0925 09:14:24.319116 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:24.319233 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 09:14:24.425300 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 10337
I20260925 09:14:24.448165 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 10479
I20260925 09:14:24.462093 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:24.448905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.3.19.67:35667)
0925 09:14:24.449382 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:24.449387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:24.449404 (+ 17us) server_negotiation.cc:407] Connection header received
0925 09:14:24.449459 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:24.449463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:24.449526 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:24.449618 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:24.451132 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:24.451819 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:24.453336 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:24.453755 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:24.453856 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:24.456587 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:24.456601 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:24.456618 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:24.456656 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:24.459390 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:24.460001 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:24.460008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:24.460010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:24.460135 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:24.460436 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:24.460439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:24.460441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:24.460793 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:24.461004 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:24.461528 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 09:14:24.461790 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":232,"threads_started":1}
I20260925 09:14:24.462769 10602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:24.555351 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 10479
I20260925 09:14:24.572839 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 9813
I20260925 09:14:24.574000 9981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:24.701553 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 9813
I20260925 09:14:24.723431 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 9912
I20260925 09:14:24.724581 10171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:24.847280 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 9912
2026-09-25T09:14:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12036 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:14:24 dist-test-slave-l6qh krb5kdc[10752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:14:24 dist-test-slave-l6qh krb5kdc[10752](info): set up 2 sockets
Sep 25 09:14:24 dist-test-slave-l6qh krb5kdc[10752](info): commencing operation
krb5kdc: starting...
W20260925 09:14:25.956394 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:14:26 dist-test-slave-l6qh krb5kdc[10752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:14:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.126.keytab.
Entry for principal oryx/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.126.keytab.
I20260925 09:14:26.197502 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:37977
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.126.keytab
--principal=oryx/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:37977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:26.527597 10765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:26.527896 10765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:26.528062 10765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:26.535539 10765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:26.535662 10765 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 09:14:26.535751 10765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:26.535851 10765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:26.535914 10765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:26.536002 10765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:14:26.546197 10765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:37977
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.126.keytab
--principal=oryx/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:37977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:26.547578 10765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:26.549065 10765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:26.561602 10771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:26.561717 10773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:26.561683 10770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:26.562680 10765 server_base.cc:1034] running on GCE node
I20260925 09:14:26.563478 10765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:26.564987 10765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:26.566298 10765 hybrid_clock.cc:648] HybridClock initialized: now 1790327666566187 us; error 57 us; skew 500 ppm
I20260925 09:14:26.566658 10765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:26 dist-test-slave-l6qh krb5kdc[10752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327666, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:26.575784 10765 init.cc:377] Logged in from keytab as oryx/127.3.19.126@KRBTEST.COM (short username oryx)
I20260925 09:14:26.577630 10765 webserver.cc:460] Webserver started at http://127.3.19.126:42757/ using document root <none> and password file <none>
I20260925 09:14:26.578420 10765 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:26.578573 10765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:26.578935 10765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:26.582139 10765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "138ace9539dd42249048a32a0566d807"
format_stamp: "Formatted at 2026-09-25 09:14:26 on dist-test-slave-l6qh"
server_key: "57dbb8896862c14cb0be008402fef29f"
server_key_iv: "782571fa7569f75daa4aecaf8acd900b"
server_key_version: "encryptionkey@0"
I20260925 09:14:26.582971 10765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "138ace9539dd42249048a32a0566d807"
format_stamp: "Formatted at 2026-09-25 09:14:26 on dist-test-slave-l6qh"
server_key: "57dbb8896862c14cb0be008402fef29f"
server_key_iv: "782571fa7569f75daa4aecaf8acd900b"
server_key_version: "encryptionkey@0"
I20260925 09:14:26.588359 10765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:14:26.591962 10780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:26.593456 10765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:26.593660 10765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "138ace9539dd42249048a32a0566d807"
format_stamp: "Formatted at 2026-09-25 09:14:26 on dist-test-slave-l6qh"
server_key: "57dbb8896862c14cb0be008402fef29f"
server_key_iv: "782571fa7569f75daa4aecaf8acd900b"
server_key_version: "encryptionkey@0"
I20260925 09:14:26.593884 10765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:26.631135 10765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:26.635532 10765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:26.635821 10765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:26.659884 10765 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:37977
I20260925 09:14:26.659891 10841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:37977 every 8 connection(s)
I20260925 09:14:26.661451 10765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:14:26.665763 10842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:26.671623 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 10765
I20260925 09:14:26.671838 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:14:26.672703 3149 external_mini_cluster.cc:1428] Setting key 7df192a34248eb669a942aae28d4d8b5
I20260925 09:14:26.677763 10842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807: Bootstrap starting.
I20260925 09:14:26.680908 10842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:26.682030 10842 log.cc:824] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:26.685503 10842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807: No bootstrap required, opened a new log
Sep 25 09:14:26 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:26.695313 10842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138ace9539dd42249048a32a0566d807" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37977 } }
I20260925 09:14:26.695827 10842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:26.695952 10842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138ace9539dd42249048a32a0566d807, State: Initialized, Role: FOLLOWER
I20260925 09:14:26.696585 10842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138ace9539dd42249048a32a0566d807" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37977 } }
I20260925 09:14:26.696877 10842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:14:26.696985 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:26.675116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.3.19.126:37977)
0925 09:14:26.675963 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:26.675976 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:26.676025 (+ 49us) server_negotiation.cc:407] Connection header received
0925 09:14:26.677819 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:26.677867 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:26.678311 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:26.678694 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:26.680141 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:26.681569 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:26.683114 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:26.684097 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:26.684301 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:26.689032 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:26.689041 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:26.689062 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:26.689109 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:26.692315 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:26.693190 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:26.693197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:26.693204 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:26.693456 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:26.693920 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:26.693933 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:26.693935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:26.694526 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:26.694750 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:26.695215 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 09:14:26.695607 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":392,"threads_started":1}
I20260925 09:14:26.697042 10842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:14:26.697311 10842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:26.700359 10842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138ace9539dd42249048a32a0566d807" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37977 } }
I20260925 09:14:26.700817 10842 leader_election.cc:302] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 138ace9539dd42249048a32a0566d807; no voters:
I20260925 09:14:26.701419 10842 leader_election.cc:288] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:14:26.701555 10847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:26.702292 10847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [term 1 LEADER]: Becoming Leader. State: Replica: 138ace9539dd42249048a32a0566d807, State: Running, Role: LEADER
I20260925 09:14:26.702920 10847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138ace9539dd42249048a32a0566d807" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37977 } }
I20260925 09:14:26.703727 10842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:26.706068 10849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 138ace9539dd42249048a32a0566d807. Latest consensus state: current_term: 1 leader_uuid: "138ace9539dd42249048a32a0566d807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138ace9539dd42249048a32a0566d807" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37977 } } }
I20260925 09:14:26.706105 10848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "138ace9539dd42249048a32a0566d807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138ace9539dd42249048a32a0566d807" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37977 } } }
I20260925 09:14:26.706467 10849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:26.706473 10848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:26.714272 10854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:14:26.719084 10854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:14:26.727797 10854 catalog_manager.cc:1347] Generated new cluster ID: 5cc81478014d43418456179b98ae5e41
I20260925 09:14:26.727926 10854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:14:26.743643 10854 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:14:26.745548 10854 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:14:26.755086 10854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807: Generated new TSK 0
I20260925 09:14:26.755988 10854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab.
Entry for principal oryx/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab.
I20260925 09:14:26.825014 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:37977
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab
--principal=oryx/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:27.160599 10868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:27.160888 10868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:27.160986 10868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:27.168325 10868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:27.168464 10868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:27.168556 10868 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 09:14:27.168638 10868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:14:27.178658 10868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab
--principal=oryx/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.19.126:37977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:27.179904 10868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:27.181129 10868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:27.194365 10874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:27.194605 10876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:27.194422 10873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:27.195755 10868 server_base.cc:1034] running on GCE node
I20260925 09:14:27.196480 10868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:27.197647 10868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:27.198904 10868 hybrid_clock.cc:648] HybridClock initialized: now 1790327667198817 us; error 58 us; skew 500 ppm
I20260925 09:14:27.199241 10868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:27 dist-test-slave-l6qh krb5kdc[10752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327667, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:27.202533 10868 init.cc:377] Logged in from keytab as oryx/127.3.19.65@KRBTEST.COM (short username oryx)
I20260925 09:14:27.204080 10868 webserver.cc:460] Webserver started at http://127.3.19.65:42365/ using document root <none> and password file <none>
I20260925 09:14:27.204859 10868 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:27.204998 10868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:27.205382 10868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:27.208504 10868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9"
format_stamp: "Formatted at 2026-09-25 09:14:27 on dist-test-slave-l6qh"
server_key: "7d3088fcbd18fc210ab94a677f887a4b"
server_key_iv: "6e1b0864595d6b4a727853bb777ebb9e"
server_key_version: "encryptionkey@0"
I20260925 09:14:27.209446 10868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9"
format_stamp: "Formatted at 2026-09-25 09:14:27 on dist-test-slave-l6qh"
server_key: "7d3088fcbd18fc210ab94a677f887a4b"
server_key_iv: "6e1b0864595d6b4a727853bb777ebb9e"
server_key_version: "encryptionkey@0"
I20260925 09:14:27.214797 10868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 09:14:27.218752 10883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:27.220273 10868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:14:27.220456 10868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9"
format_stamp: "Formatted at 2026-09-25 09:14:27 on dist-test-slave-l6qh"
server_key: "7d3088fcbd18fc210ab94a677f887a4b"
server_key_iv: "6e1b0864595d6b4a727853bb777ebb9e"
server_key_version: "encryptionkey@0"
I20260925 09:14:27.220696 10868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:27.239132 10868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:27.243605 10868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:27.243947 10868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:27.247088 10868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:27.247207 10868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:27.247406 10868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:27.247561 10868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 09:14:27 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327667, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:27.270829 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:27.247260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:49953 (local address 127.3.19.126:37977)
0925 09:14:27.247849 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:27.247856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:27.249119 (+ 1263us) server_negotiation.cc:407] Connection header received
0925 09:14:27.251132 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:27.251143 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:27.251262 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:27.251419 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:27.253537 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.254587 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.257179 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.257749 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.257894 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:27.263383 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:27.263392 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:27.263397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:27.263472 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:27.266565 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:27.267967 (+ 1402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:27.267971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:27.267974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:27.268140 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:27.268904 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:27.268909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:27.268913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:27.269318 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:27.269479 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:27.270220 (+ 741us) server_negotiation.cc:299] Negotiation successful
0925 09:14:27.270447 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":296,"threads_started":1}
I20260925 09:14:27.271888 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:27.247873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:37977 (local address 127.3.19.65:49953)
0925 09:14:27.248820 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:27.248868 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:27.250846 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:27.251641 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:27.251648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:27.252390 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:27.253335 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:27.253369 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.254804 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.254813 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:27.256943 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:27.256962 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.258082 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.258094 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:27.258535 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:27.259530 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:27.259566 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:27.263048 (+ 3482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:27.266752 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:27.266769 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:27.266792 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:27.267782 (+ 990us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:27.268290 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:27.268297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:27.268301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:27.268736 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:27.269456 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:27.269465 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:27.269845 (+ 380us) client_negotiation.cc:769] Sending connection context
0925 09:14:27.270121 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 09:14:27.270691 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":423,"threads_started":1}
I20260925 09:14:27.296130 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:27.279777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:37977 (local address 127.3.19.65:47497)
0925 09:14:27.282076 (+ 2299us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:27.282094 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:27.282291 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:27.282646 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:27.282651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:27.283133 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:27.283425 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:27.283434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.284410 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.284417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:27.285591 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:27.285608 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.286614 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.286628 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:27.286906 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:27.287684 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:27.287721 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:27.288676 (+ 955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:27.292716 (+ 4040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:27.292727 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:27.292731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:27.293444 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:27.293892 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:27.293898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:27.293901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:27.294036 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:27.294770 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:27.294774 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:27.295056 (+ 282us) client_negotiation.cc:769] Sending connection context
0925 09:14:27.295230 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 09:14:27.295605 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2123,"thread_start_us":1904,"threads_started":1}
I20260925 09:14:27.296929 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:27.279358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:47497 (local address 127.3.19.126:37977)
0925 09:14:27.279597 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:27.279605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:27.282206 (+ 2601us) server_negotiation.cc:407] Connection header received
0925 09:14:27.282430 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:27.282436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:27.282509 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:27.282619 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:27.283583 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.284280 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.285806 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.286368 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.286514 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:27.288894 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:27.288903 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:27.288910 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:27.288958 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:27.292560 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:27.293638 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:27.293642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:27.293644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:27.293775 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:27.294158 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:27.294162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:27.294166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:27.294569 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:27.294718 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:27.296528 (+ 1810us) server_negotiation.cc:299] Negotiation successful
0925 09:14:27.296715 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 09:14:27.306365 10868 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:33377
I20260925 09:14:27.306393 11004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:33377 every 8 connection(s)
I20260925 09:14:27.307771 10868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:14:27.308329 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 10868
I20260925 09:14:27.308568 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:14:27.309484 3149 external_mini_cluster.cc:1428] Setting key 571aa2d69732d60b2093604d55a25061
I20260925 09:14:27.310366 11005 heartbeater.cc:347] Connected to a master server at 127.3.19.126:37977
I20260925 09:14:27.310649 11005 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:27.311426 11005 heartbeater.cc:511] Master 127.3.19.126:37977 requested a full tablet report, sending...
I20260925 09:14:27.313426 10797 ts_manager.cc:194] Registered new tserver with Master: ee4f2a8bf0444ccb9e2e14bec877ddc9 (127.3.19.65:33377)
I20260925 09:14:27.314854 10797 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:49953
WARNING: no policy specified for oryx/127.3.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab.
Entry for principal oryx/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab.
I20260925 09:14:27.360008 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:37977
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab
--principal=oryx/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:27.693914 11009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:27.694242 11009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:27.694415 11009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:27.702031 11009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:27.702186 11009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:27.702339 11009 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 09:14:27.702484 11009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:14:27.712752 11009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab
--principal=oryx/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.19.126:37977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:27.714166 11009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:27.715525 11009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:27.728793 11014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:27.728857 11017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:27.728892 11015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:27.730232 11009 server_base.cc:1034] running on GCE node
I20260925 09:14:27.730954 11009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:27.731916 11009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:27.733186 11009 hybrid_clock.cc:648] HybridClock initialized: now 1790327667733099 us; error 62 us; skew 500 ppm
I20260925 09:14:27.733567 11009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:27 dist-test-slave-l6qh krb5kdc[10752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327667, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:27.736726 11009 init.cc:377] Logged in from keytab as oryx/127.3.19.66@KRBTEST.COM (short username oryx)
I20260925 09:14:27.738303 11009 webserver.cc:460] Webserver started at http://127.3.19.66:40769/ using document root <none> and password file <none>
I20260925 09:14:27.739022 11009 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:27.739154 11009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:27.739498 11009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:27.742587 11009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "c539aaf8bdec43728f8e8870d05f91d6"
format_stamp: "Formatted at 2026-09-25 09:14:27 on dist-test-slave-l6qh"
server_key: "e15dbc9a007f82c47f69f9f9e4def81c"
server_key_iv: "e919d890d164e120a3d5b5a123aa5349"
server_key_version: "encryptionkey@0"
I20260925 09:14:27.743399 11009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "c539aaf8bdec43728f8e8870d05f91d6"
format_stamp: "Formatted at 2026-09-25 09:14:27 on dist-test-slave-l6qh"
server_key: "e15dbc9a007f82c47f69f9f9e4def81c"
server_key_iv: "e919d890d164e120a3d5b5a123aa5349"
server_key_version: "encryptionkey@0"
I20260925 09:14:27.748800 11009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 09:14:27.752575 11025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:27.754139 11009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:27.754324 11009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "c539aaf8bdec43728f8e8870d05f91d6"
format_stamp: "Formatted at 2026-09-25 09:14:27 on dist-test-slave-l6qh"
server_key: "e15dbc9a007f82c47f69f9f9e4def81c"
server_key_iv: "e919d890d164e120a3d5b5a123aa5349"
server_key_version: "encryptionkey@0"
I20260925 09:14:27.754565 11009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:27.778505 11009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:27.783239 11009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:27.783560 11009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:27.786633 11009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:27.786761 11009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:27.787003 11009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:27.787113 11009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 09:14:27 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327667, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:27.811785 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:27.787113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:47689 (local address 127.3.19.126:37977)
0925 09:14:27.787326 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:27.787333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:27.788889 (+ 1556us) server_negotiation.cc:407] Connection header received
0925 09:14:27.790947 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:27.790953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:27.791056 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:27.791168 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:27.793215 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.794311 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.796928 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.797484 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.797625 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:27.803702 (+ 6077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:27.803706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:27.803710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:27.803745 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:27.807023 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:27.808353 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:27.808361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:27.808366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:27.808532 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:27.809270 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:27.809294 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:27.809301 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:27.809871 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:27.810088 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:27.810856 (+ 768us) server_negotiation.cc:299] Negotiation successful
0925 09:14:27.811205 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 09:14:27.812357 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:27.787738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:37977 (local address 127.3.19.66:47689)
0925 09:14:27.788683 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:27.788733 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:27.790647 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:27.791405 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:27.791414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:27.792100 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:27.793034 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:27.793065 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.794451 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.794457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:27.796780 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:27.796813 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.797744 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.797755 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:27.798127 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:27.799134 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:27.799178 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:27.803388 (+ 4210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:27.807207 (+ 3819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:27.807224 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:27.807248 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:27.808192 (+ 944us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:27.808656 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:27.808665 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:27.808672 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:27.809056 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:27.810062 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:27.810068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:27.810427 (+ 359us) client_negotiation.cc:769] Sending connection context
0925 09:14:27.810696 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 09:14:27.811100 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":403,"threads_started":1}
I20260925 09:14:27.833766 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:27.818227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:42451 (local address 127.3.19.126:37977)
0925 09:14:27.818398 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:27.818402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:27.819143 (+ 741us) server_negotiation.cc:407] Connection header received
0925 09:14:27.819358 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:27.819362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:27.819421 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:27.819532 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:27.820454 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.821195 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.823463 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.824002 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.824120 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:27.826580 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:27.826587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:27.826591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:27.826629 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:27.830076 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:27.831017 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:27.831024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:27.831027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:27.831223 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:27.831825 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:27.831835 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:27.831841 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:27.832426 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:27.832652 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:27.833163 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 09:14:27.833418 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 09:14:27.833953 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:27.818347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:37977 (local address 127.3.19.66:42451)
0925 09:14:27.819035 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:27.819051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:27.819232 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:27.819552 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:27.819556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:27.820036 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:27.820315 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:27.820323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.821425 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.821429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:27.823251 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:27.823266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:27.824471 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:27.824476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:27.824683 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:27.825408 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:27.825425 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:27.826393 (+ 968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:27.830218 (+ 3825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:27.830224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:27.830228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:27.830890 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:27.831371 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:27.831379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:27.831390 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:27.831613 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:27.832606 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:27.832614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:27.832871 (+ 257us) client_negotiation.cc:769] Sending connection context
0925 09:14:27.833072 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 09:14:27.833471 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":425,"threads_started":1}
I20260925 09:14:27.848981 11009 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:37839
I20260925 09:14:27.849010 11145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:37839 every 8 connection(s)
I20260925 09:14:27.850402 11009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:14:27.852875 11146 heartbeater.cc:347] Connected to a master server at 127.3.19.126:37977
I20260925 09:14:27.853089 11146 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:27.853649 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 11009
I20260925 09:14:27.853832 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:14:27.854007 11146 heartbeater.cc:511] Master 127.3.19.126:37977 requested a full tablet report, sending...
I20260925 09:14:27.854362 3149 external_mini_cluster.cc:1428] Setting key cb7796b02a55a8ee5543d3d3cef4d236
I20260925 09:14:27.855638 10797 ts_manager.cc:194] Registered new tserver with Master: c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66:37839)
I20260925 09:14:27.856768 10797 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:47689
WARNING: no policy specified for oryx/127.3.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab.
Entry for principal oryx/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab.
I20260925 09:14:27.904347 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:37977
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab
--principal=oryx/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:28.239732 11150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:28.240100 11150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:28.240280 11150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:28.247541 11150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:28.247716 11150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:28.247887 11150 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 09:14:28.248044 11150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:14:28.258774 11150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab
--principal=oryx/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.19.126:37977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:28.260231 11150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:28.261672 11150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:28.275178 11156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:28.275192 11158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:28.275864 11155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:28.276126 11150 server_base.cc:1034] running on GCE node
I20260925 09:14:28.276783 11150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:28.277890 11150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:28.279155 11150 hybrid_clock.cc:648] HybridClock initialized: now 1790327668279069 us; error 63 us; skew 500 ppm
I20260925 09:14:28.279497 11150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:28 dist-test-slave-l6qh krb5kdc[10752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327668, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:28.282853 11150 init.cc:377] Logged in from keytab as oryx/127.3.19.67@KRBTEST.COM (short username oryx)
I20260925 09:14:28.284386 11150 webserver.cc:460] Webserver started at http://127.3.19.67:40269/ using document root <none> and password file <none>
I20260925 09:14:28.285118 11150 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:28.285295 11150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:28.285652 11150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:28.288655 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "f5fc6ff41f7a4628be2bf3ba26da9356"
format_stamp: "Formatted at 2026-09-25 09:14:28 on dist-test-slave-l6qh"
server_key: "81860803f8f035eb7f2de80238131bde"
server_key_iv: "8e21c42527b5100e0023677f46d05c66"
server_key_version: "encryptionkey@0"
I20260925 09:14:28.289544 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "f5fc6ff41f7a4628be2bf3ba26da9356"
format_stamp: "Formatted at 2026-09-25 09:14:28 on dist-test-slave-l6qh"
server_key: "81860803f8f035eb7f2de80238131bde"
server_key_iv: "8e21c42527b5100e0023677f46d05c66"
server_key_version: "encryptionkey@0"
I20260925 09:14:28.294822 11150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 09:14:28.298594 11165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:28.300153 11150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 09:14:28.300367 11150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "f5fc6ff41f7a4628be2bf3ba26da9356"
format_stamp: "Formatted at 2026-09-25 09:14:28 on dist-test-slave-l6qh"
server_key: "81860803f8f035eb7f2de80238131bde"
server_key_iv: "8e21c42527b5100e0023677f46d05c66"
server_key_version: "encryptionkey@0"
I20260925 09:14:28.300637 11150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:28.313680 11150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:28.317680 11005 heartbeater.cc:503] Master 127.3.19.126:37977 was elected leader, sending a full tablet report...
I20260925 09:14:28.319182 11150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:28.319468 11150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:28.322381 11150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:28.322494 11150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:28.322634 11150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:28.322744 11150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 25 09:14:28 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327668, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:28.348671 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.322724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:57001 (local address 127.3.19.126:37977)
0925 09:14:28.322919 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:28.322923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:28.324505 (+ 1582us) server_negotiation.cc:407] Connection header received
0925 09:14:28.326815 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:28.326820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:28.326905 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:28.327034 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:28.328576 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.329708 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.333297 (+ 3589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.333885 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.334055 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.339687 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:28.339698 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:28.339706 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:28.339754 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:28.344256 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:28.345412 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:28.345417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:28.345419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:28.345620 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:28.346420 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:28.346427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:28.346432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:28.346749 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:28.346940 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:28.348204 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0925 09:14:28.348404 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 09:14:28.350942 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.323461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:37977 (local address 127.3.19.67:57001)
0925 09:14:28.324341 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:28.324384 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:28.326512 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:28.327299 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:28.327307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:28.327806 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:28.328399 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.328417 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.329935 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.329941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.333069 (+ 3128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.333093 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.334188 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.334195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.334533 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.335520 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:28.335547 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:28.339290 (+ 3743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:28.344455 (+ 5165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:28.344472 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:28.344498 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:28.345202 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:28.345787 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:28.345794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:28.345800 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:28.346249 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:28.346930 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:28.346943 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:28.347627 (+ 684us) client_negotiation.cc:769] Sending connection context
0925 09:14:28.348065 (+ 438us) client_negotiation.cc:240] Negotiation successful
0925 09:14:28.348860 (+ 795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":419,"threads_started":1}
I20260925 09:14:28.371765 11150 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:33599
I20260925 09:14:28.371824 11285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:33599 every 8 connection(s)
I20260925 09:14:28.372764 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.357081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:32907 (local address 127.3.19.126:37977)
0925 09:14:28.357365 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:28.357374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:28.357878 (+ 504us) server_negotiation.cc:407] Connection header received
0925 09:14:28.358101 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:28.358109 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:28.358187 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:28.358268 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:28.359801 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.360691 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.362865 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.363300 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.363446 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.366145 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:28.366156 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:28.366159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:28.366198 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:28.369862 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:28.370590 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:28.370595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:28.370599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:28.370765 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:28.371266 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:28.371271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:28.371274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:28.371668 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:28.371835 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:28.372301 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 09:14:28.372514 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 09:14:28.373111 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.356958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:37977 (local address 127.3.19.67:32907)
0925 09:14:28.357740 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:28.357772 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:28.357951 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:28.358316 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:28.358323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:28.359091 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:28.359613 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.359630 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.360859 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.360875 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.362712 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.362725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.363564 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.363571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.363736 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.364570 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:28.364599 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:28.365868 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:28.370006 (+ 4138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:28.370014 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:28.370020 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:28.370442 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:28.370929 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:28.370935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:28.370939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:28.371141 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:28.371812 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:28.371817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:28.372040 (+ 223us) client_negotiation.cc:769] Sending connection context
0925 09:14:28.372203 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 09:14:28.372554 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":412,"threads_started":1}
I20260925 09:14:28.373593 11150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:14:28.375689 11286 heartbeater.cc:347] Connected to a master server at 127.3.19.126:37977
I20260925 09:14:28.375926 11286 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:28.376760 11286 heartbeater.cc:511] Master 127.3.19.126:37977 requested a full tablet report, sending...
I20260925 09:14:28.378290 10797 ts_manager.cc:194] Registered new tserver with Master: f5fc6ff41f7a4628be2bf3ba26da9356 (127.3.19.67:33599)
I20260925 09:14:28.378629 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 11150
I20260925 09:14:28.378804 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:14:28.379243 3149 external_mini_cluster.cc:1428] Setting key abac2229d2da1fc15507c228123931f4
I20260925 09:14:28.379242 10797 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:57001
I20260925 09:14:28.382222 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 09:14:28 dist-test-slave-l6qh krb5kdc[10752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327668, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 09:14:28 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:28 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 09:14:28.436831 10891 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:28.425683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.3.19.126:37977)
0925 09:14:28.425915 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:28.425922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:28.426140 (+ 218us) server_negotiation.cc:407] Connection header received
0925 09:14:28.426288 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:28.426292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:28.426357 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:28.426443 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:28.427753 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.428496 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.429881 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.430290 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.430386 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.436568 (+ 6182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49256: BlockingRecv error: recv got EOF from 127.0.0.1:49256 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 09:14:28 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327668, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:28.454185 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.439815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.3.19.126:37977)
0925 09:14:28.440047 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:28.440055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:28.440338 (+ 283us) server_negotiation.cc:407] Connection header received
0925 09:14:28.440476 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:28.440480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:28.440594 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:28.440686 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:28.441933 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.442770 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.444067 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.444555 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.444676 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.448806 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:28.448812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:28.448815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:28.448856 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:28.451518 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:28.452166 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:28.452170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:28.452173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:28.452316 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:28.452645 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:28.452648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:28.452650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:28.453037 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:28.453416 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:28.453680 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 09:14:28.453859 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 09:14:28.458439 10797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:14:28.465337 10797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:14:28.490434 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.478681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34418 (local address 127.3.19.66:37839)
0925 09:14:28.479539 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:28.479546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:28.479790 (+ 244us) server_negotiation.cc:407] Connection header received
0925 09:14:28.480099 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:28.480107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:28.480382 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:28.480597 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:28.482616 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.484208 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.487657 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.489060 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.489187 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.489422 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:28.489941 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 09:14:28.490108 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":570,"threads_started":1}
I20260925 09:14:28.490505 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.478479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:37839 (local address 127.0.0.1:34418)
0925 09:14:28.479637 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:28.479681 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:28.479869 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:28.480845 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:28.480852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:28.480880 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:28.482109 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.482127 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.484408 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.484416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.487478 (+ 3062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.487493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.489562 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.489567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.489745 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.489798 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 09:14:28.489884 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 09:14:28.490093 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":217,"thread_start_us":747,"threads_started":1}
I20260925 09:14:28.491786 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.480323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51530 (local address 127.3.19.65:33377)
0925 09:14:28.481520 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:28.481531 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:28.481564 (+ 33us) server_negotiation.cc:407] Connection header received
0925 09:14:28.481673 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:28.481680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:28.481957 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:28.482401 (+ 444us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:28.483047 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.484361 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.487721 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.489056 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.489186 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.489425 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:28.491333 (+ 1908us) server_negotiation.cc:299] Negotiation successful
0925 09:14:28.491509 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":905,"threads_started":1}
I20260925 09:14:28.492215 11309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.481515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.3.19.67:33599)
0925 09:14:28.482145 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:28.482156 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:28.483521 (+ 1365us) server_negotiation.cc:407] Connection header received
0925 09:14:28.483656 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:28.483662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:28.483883 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:28.484059 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:28.484984 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.486691 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.488866 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.490442 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.490575 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.490845 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:28.491635 (+ 790us) server_negotiation.cc:299] Negotiation successful
0925 09:14:28.491841 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":675,"threads_started":1}
I20260925 09:14:28.492713 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.480895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:33599 (local address 127.0.0.1:54038)
0925 09:14:28.483379 (+ 2484us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:28.483398 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:28.483531 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:28.484070 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:28.484075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:28.484101 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:28.484689 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.484705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.486842 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.486848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.488689 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.488701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.491003 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.491013 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.491241 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.491282 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 09:14:28.491366 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 09:14:28.491476 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2363,"mutex_wait_us":1,"thread_start_us":2729,"threads_started":1}
I20260925 09:14:28.495096 11079 tablet_service.cc:1467] Processing CreateTablet for tablet 067936082e1d4b19a3ec2a4bcf19b13c (DEFAULT_TABLE table=test-table [id=7273ab7ee9d24ed589f78e6bde5eb335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:28.496765 11079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067936082e1d4b19a3ec2a4bcf19b13c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:28.497280 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.480077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:33377 (local address 127.0.0.1:51530)
0925 09:14:28.480737 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:28.480756 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:28.480858 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:28.482240 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:28.482248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:28.482283 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:28.482873 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.482885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.484506 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.484512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.487517 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.487529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.489303 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.489310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.489529 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.489662 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 09:14:28.491263 (+ 1601us) client_negotiation.cc:240] Negotiation successful
0925 09:14:28.491422 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":99,"thread_start_us":298,"threads_started":1}
I20260925 09:14:28.496630 11215 tablet_service.cc:1467] Processing CreateTablet for tablet 067936082e1d4b19a3ec2a4bcf19b13c (DEFAULT_TABLE table=test-table [id=7273ab7ee9d24ed589f78e6bde5eb335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:28.501396 11215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067936082e1d4b19a3ec2a4bcf19b13c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:28.501020 10934 tablet_service.cc:1467] Processing CreateTablet for tablet 067936082e1d4b19a3ec2a4bcf19b13c (DEFAULT_TABLE table=test-table [id=7273ab7ee9d24ed589f78e6bde5eb335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:28.502528 10934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067936082e1d4b19a3ec2a4bcf19b13c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:28.513336 11310 tablet_bootstrap.cc:492] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356: Bootstrap starting.
I20260925 09:14:28.515455 11311 tablet_bootstrap.cc:492] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6: Bootstrap starting.
I20260925 09:14:28.517508 11310 tablet_bootstrap.cc:654] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:28.518832 11311 tablet_bootstrap.cc:654] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:28.518970 11310 log.cc:824] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:28.520160 11311 log.cc:824] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:28.520457 11312 tablet_bootstrap.cc:492] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9: Bootstrap starting.
I20260925 09:14:28.523018 11310 tablet_bootstrap.cc:492] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356: No bootstrap required, opened a new log
I20260925 09:14:28.523372 11310 ts_tablet_manager.cc:1397] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260925 09:14:28.526546 11311 tablet_bootstrap.cc:492] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6: No bootstrap required, opened a new log
I20260925 09:14:28.526850 11311 ts_tablet_manager.cc:1397] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260925 09:14:28.527185 11312 tablet_bootstrap.cc:654] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:28.528622 11312 log.cc:824] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:28.531875 11310 raft_consensus.cc:348] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:28.532292 11310 raft_consensus.cc:374] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:28.532389 11310 raft_consensus.cc:729] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fc6ff41f7a4628be2bf3ba26da9356, State: Initialized, Role: FOLLOWER
I20260925 09:14:28.532846 11312 tablet_bootstrap.cc:492] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9: No bootstrap required, opened a new log
I20260925 09:14:28.533172 11312 ts_tablet_manager.cc:1397] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260925 09:14:28.533087 11310 consensus_queue.cc:260] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:28.534302 11286 heartbeater.cc:503] Master 127.3.19.126:37977 was elected leader, sending a full tablet report...
I20260925 09:14:28.535040 11310 ts_tablet_manager.cc:1428] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260925 09:14:28.538748 11311 raft_consensus.cc:348] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:28.539304 11311 raft_consensus.cc:374] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:28.539434 11311 raft_consensus.cc:729] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c539aaf8bdec43728f8e8870d05f91d6, State: Initialized, Role: FOLLOWER
I20260925 09:14:28.540144 11311 consensus_queue.cc:260] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:28.541347 11146 heartbeater.cc:503] Master 127.3.19.126:37977 was elected leader, sending a full tablet report...
I20260925 09:14:28.542187 11311 ts_tablet_manager.cc:1428] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260925 09:14:28.544816 11312 raft_consensus.cc:348] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:28.545228 11312 raft_consensus.cc:374] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:28.545384 11312 raft_consensus.cc:729] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4f2a8bf0444ccb9e2e14bec877ddc9, State: Initialized, Role: FOLLOWER
I20260925 09:14:28.545993 11312 consensus_queue.cc:260] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:28.547857 11312 ts_tablet_manager.cc:1428] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
W20260925 09:14:28.560230 11006 tablet.cc:2369] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:14:28.602592 11147 tablet.cc:2369] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:14:28.625653 11287 tablet.cc:2369] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:14:28.911953 11316 raft_consensus.cc:482] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:14:28.912231 11316 raft_consensus.cc:504] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:28.913902 11316 leader_election.cc:288] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66:37839), ee4f2a8bf0444ccb9e2e14bec877ddc9 (127.3.19.65:33377)
I20260925 09:14:28.919797 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.914318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:55557 (local address 127.3.19.66:37839)
0925 09:14:28.914514 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:28.914520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:28.914844 (+ 324us) server_negotiation.cc:407] Connection header received
0925 09:14:28.914939 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:28.914944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:28.915019 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:28.915143 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:28.915781 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.916486 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.918158 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.918881 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.919033 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.919144 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:28.919414 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 09:14:28.919536 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 09:14:28.919857 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.914127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:37839 (local address 127.3.19.67:55557)
0925 09:14:28.914619 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:28.914639 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:28.914779 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:28.915325 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:28.915329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:28.915355 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:28.915611 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.915630 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.916621 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.916629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.917982 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.917993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.919047 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.919052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.919263 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.919287 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 09:14:28.919389 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 09:14:28.919516 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":248,"threads_started":1}
I20260925 09:14:28.920710 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.915025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:49705 (local address 127.3.19.65:33377)
0925 09:14:28.915285 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:28.915294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:28.915858 (+ 564us) server_negotiation.cc:407] Connection header received
0925 09:14:28.916039 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:28.916044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:28.916124 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:28.916218 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:28.916664 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.917420 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.918959 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.919743 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.919896 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.919999 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:28.920348 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 09:14:28.920470 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 09:14:28.920747 11320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:28.914785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:33377 (local address 127.3.19.67:49705)
0925 09:14:28.915752 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:28.915770 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:28.915901 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:28.916231 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:28.916235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:28.916260 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:28.916515 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.916523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.917551 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.917558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.918809 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:28.918819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:28.919959 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:28.919965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:28.920192 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:28.920225 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 09:14:28.920324 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 09:14:28.920447 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":686,"threads_started":1}
I20260925 09:14:28.920919 11101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067936082e1d4b19a3ec2a4bcf19b13c" candidate_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c539aaf8bdec43728f8e8870d05f91d6" is_pre_election: true
I20260925 09:14:28.921629 11101 raft_consensus.cc:2454] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5fc6ff41f7a4628be2bf3ba26da9356 in term 0.
I20260925 09:14:28.921855 10960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067936082e1d4b19a3ec2a4bcf19b13c" candidate_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" is_pre_election: true
I20260925 09:14:28.922341 11169 leader_election.cc:302] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c539aaf8bdec43728f8e8870d05f91d6, f5fc6ff41f7a4628be2bf3ba26da9356; no voters:
I20260925 09:14:28.922390 10960 raft_consensus.cc:2454] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5fc6ff41f7a4628be2bf3ba26da9356 in term 0.
I20260925 09:14:28.922724 11316 raft_consensus.cc:2781] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:14:28.922902 11316 raft_consensus.cc:482] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:14:28.923019 11316 raft_consensus.cc:3037] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:28.925447 11316 raft_consensus.cc:504] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:28.926061 11316 leader_election.cc:288] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [CANDIDATE]: Term 1 election: Requested vote from peers c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66:37839), ee4f2a8bf0444ccb9e2e14bec877ddc9 (127.3.19.65:33377)
I20260925 09:14:28.926546 11101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067936082e1d4b19a3ec2a4bcf19b13c" candidate_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c539aaf8bdec43728f8e8870d05f91d6"
I20260925 09:14:28.926575 10960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067936082e1d4b19a3ec2a4bcf19b13c" candidate_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9"
I20260925 09:14:28.926780 10960 raft_consensus.cc:3037] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:28.926765 11101 raft_consensus.cc:3037] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:28.928977 10960 raft_consensus.cc:2454] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5fc6ff41f7a4628be2bf3ba26da9356 in term 1.
I20260925 09:14:28.928972 11101 raft_consensus.cc:2454] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5fc6ff41f7a4628be2bf3ba26da9356 in term 1.
I20260925 09:14:28.929447 11169 leader_election.cc:302] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c539aaf8bdec43728f8e8870d05f91d6, f5fc6ff41f7a4628be2bf3ba26da9356; no voters:
I20260925 09:14:28.929813 11316 raft_consensus.cc:2781] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:28.930258 11316 raft_consensus.cc:686] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 LEADER]: Becoming Leader. State: Replica: f5fc6ff41f7a4628be2bf3ba26da9356, State: Running, Role: LEADER
I20260925 09:14:28.930785 11316 consensus_queue.cc:237] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:28.935668 10796 catalog_manager.cc:5495] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5fc6ff41f7a4628be2bf3ba26da9356 (127.3.19.67). New cstate: current_term: 1 leader_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } health_report { overall_health: HEALTHY } } }
I20260925 09:14:29.057756 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:29.052057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.3.19.67:33599)
0925 09:14:29.052709 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:29.052714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:29.052731 (+ 17us) server_negotiation.cc:407] Connection header received
0925 09:14:29.052817 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:29.052820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:29.052902 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:29.053062 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:14:29.053501 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.054265 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.055620 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.056150 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.056253 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.056627 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:14:29.057199 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:14:29.057336 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:29.057433 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 09:14:29.057510 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":402,"threads_started":1}
I20260925 09:14:29.066237 11101 raft_consensus.cc:1261] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Refusing update from remote peer f5fc6ff41f7a4628be2bf3ba26da9356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:29.066237 10960 raft_consensus.cc:1261] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Refusing update from remote peer f5fc6ff41f7a4628be2bf3ba26da9356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:29.067274 11316 consensus_queue.cc:1035] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:29.067608 11321 consensus_queue.cc:1035] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:29.094729 10796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:29.095012 10796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:29.098694 10796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807: Sending DeleteTablet for 3 replicas of tablet 067936082e1d4b19a3ec2a4bcf19b13c
I20260925 09:14:29.099952 11079 tablet_service.cc:1514] Processing DeleteTablet for tablet 067936082e1d4b19a3ec2a4bcf19b13c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:29 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:34418
I20260925 09:14:29.100333 11215 tablet_service.cc:1514] Processing DeleteTablet for tablet 067936082e1d4b19a3ec2a4bcf19b13c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:29 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:54038
I20260925 09:14:29.100709 10934 tablet_service.cc:1514] Processing DeleteTablet for tablet 067936082e1d4b19a3ec2a4bcf19b13c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:29 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:51530
I20260925 09:14:29.101151 10796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:14:29.101939 10796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:14:29.103883 11340 tablet_replica.cc:330] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9: stopping tablet replica
I20260925 09:14:29.104702 11339 tablet_replica.cc:330] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356: stopping tablet replica
I20260925 09:14:29.105252 11339 raft_consensus.cc:2229] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:14:29.106076 11339 raft_consensus.cc:2258] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:29.106726 11340 raft_consensus.cc:2229] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:29.107323 11340 raft_consensus.cc:2258] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:29.108225 11338 tablet_replica.cc:330] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6: stopping tablet replica
I20260925 09:14:29.108654 11338 raft_consensus.cc:2229] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:29.109108 11338 raft_consensus.cc:2258] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:29.111091 11338 ts_tablet_manager.cc:1905] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:29.111822 10934 tablet_service.cc:1467] Processing CreateTablet for tablet 3d18be178e0c4916b60ac096ae5bebff (DEFAULT_TABLE table=test-table [id=7da4f9493143432a9d2c583a8ab2a169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:29.112279 10934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d18be178e0c4916b60ac096ae5bebff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:29.115355 11079 tablet_service.cc:1467] Processing CreateTablet for tablet 3d18be178e0c4916b60ac096ae5bebff (DEFAULT_TABLE table=test-table [id=7da4f9493143432a9d2c583a8ab2a169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:29.118211 11338 ts_tablet_manager.cc:1918] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:29.118376 11338 log.cc:1196] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/067936082e1d4b19a3ec2a4bcf19b13c
I20260925 09:14:29.118888 11338 ts_tablet_manager.cc:1939] T 067936082e1d4b19a3ec2a4bcf19b13c P c539aaf8bdec43728f8e8870d05f91d6: Deleting consensus metadata
I20260925 09:14:29.118904 11340 ts_tablet_manager.cc:1905] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:29.119594 11079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d18be178e0c4916b60ac096ae5bebff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:29.124624 11215 tablet_service.cc:1467] Processing CreateTablet for tablet 3d18be178e0c4916b60ac096ae5bebff (DEFAULT_TABLE table=test-table [id=7da4f9493143432a9d2c583a8ab2a169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:29.125999 11215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d18be178e0c4916b60ac096ae5bebff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:29.126377 11340 ts_tablet_manager.cc:1918] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:29.126564 11340 log.cc:1196] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/067936082e1d4b19a3ec2a4bcf19b13c
I20260925 09:14:29.127238 11340 ts_tablet_manager.cc:1939] T 067936082e1d4b19a3ec2a4bcf19b13c P ee4f2a8bf0444ccb9e2e14bec877ddc9: Deleting consensus metadata
I20260925 09:14:29.129257 10784 catalog_manager.cc:4841] TS ee4f2a8bf0444ccb9e2e14bec877ddc9 (127.3.19.65:33377): tablet 067936082e1d4b19a3ec2a4bcf19b13c (table test-table [id=7273ab7ee9d24ed589f78e6bde5eb335]) successfully deleted
I20260925 09:14:29.129700 11341 tablet_bootstrap.cc:492] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9: Bootstrap starting.
I20260925 09:14:29.131754 11341 tablet_bootstrap.cc:654] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:29.134379 11339 ts_tablet_manager.cc:1905] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:29.140053 10784 catalog_manager.cc:4841] TS c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66:37839): tablet 067936082e1d4b19a3ec2a4bcf19b13c (table test-table [id=7273ab7ee9d24ed589f78e6bde5eb335]) successfully deleted
I20260925 09:14:29.140596 11341 tablet_bootstrap.cc:492] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9: No bootstrap required, opened a new log
I20260925 09:14:29.140777 11341 ts_tablet_manager.cc:1397] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260925 09:14:29.141666 11341 raft_consensus.cc:348] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.141952 11341 raft_consensus.cc:374] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:29.142046 11341 raft_consensus.cc:729] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4f2a8bf0444ccb9e2e14bec877ddc9, State: Initialized, Role: FOLLOWER
I20260925 09:14:29.142378 11341 consensus_queue.cc:260] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.143349 11343 tablet_bootstrap.cc:492] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6: Bootstrap starting.
I20260925 09:14:29.147092 11341 ts_tablet_manager.cc:1428] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260925 09:14:29.150465 11343 tablet_bootstrap.cc:654] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:29.152151 11344 tablet_bootstrap.cc:492] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356: Bootstrap starting.
I20260925 09:14:29.152694 11339 ts_tablet_manager.cc:1918] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:29.152930 11339 log.cc:1196] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/067936082e1d4b19a3ec2a4bcf19b13c
I20260925 09:14:29.153792 11339 ts_tablet_manager.cc:1939] T 067936082e1d4b19a3ec2a4bcf19b13c P f5fc6ff41f7a4628be2bf3ba26da9356: Deleting consensus metadata
I20260925 09:14:29.154616 11344 tablet_bootstrap.cc:654] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:29.155818 10784 catalog_manager.cc:4841] TS f5fc6ff41f7a4628be2bf3ba26da9356 (127.3.19.67:33599): tablet 067936082e1d4b19a3ec2a4bcf19b13c (table test-table [id=7273ab7ee9d24ed589f78e6bde5eb335]) successfully deleted
I20260925 09:14:29.157909 11343 tablet_bootstrap.cc:492] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6: No bootstrap required, opened a new log
I20260925 09:14:29.157969 11344 tablet_bootstrap.cc:492] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356: No bootstrap required, opened a new log
I20260925 09:14:29.158093 11343 ts_tablet_manager.cc:1397] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260925 09:14:29.158164 11344 ts_tablet_manager.cc:1397] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 09:14:29.158941 11343 raft_consensus.cc:348] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.159206 11343 raft_consensus.cc:374] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:29.159307 11343 raft_consensus.cc:729] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c539aaf8bdec43728f8e8870d05f91d6, State: Initialized, Role: FOLLOWER
I20260925 09:14:29.159119 11344 raft_consensus.cc:348] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.159426 11344 raft_consensus.cc:374] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:29.159531 11344 raft_consensus.cc:729] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fc6ff41f7a4628be2bf3ba26da9356, State: Initialized, Role: FOLLOWER
I20260925 09:14:29.159607 11343 consensus_queue.cc:260] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.160231 11343 ts_tablet_manager.cc:1428] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 09:14:29.159782 11344 consensus_queue.cc:260] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.160714 11344 ts_tablet_manager.cc:1428] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 09:14:29.164672 11327 raft_consensus.cc:482] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:14:29.164927 11327 raft_consensus.cc:504] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.166898 11327 leader_election.cc:288] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66:37839), f5fc6ff41f7a4628be2bf3ba26da9356 (127.3.19.67:33599)
I20260925 09:14:29.175181 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:29.169162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:39073 (local address 127.3.19.67:33599)
0925 09:14:29.169374 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:29.169382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:29.169401 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:14:29.169699 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:29.169702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:29.169758 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:29.169856 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:29.170370 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.171153 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.173050 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.174156 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.174293 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.174408 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:29.174755 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 09:14:29.174879 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 09:14:29.175315 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:29.168701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:33599 (local address 127.3.19.65:39073)
0925 09:14:29.169318 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:29.169335 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:29.169494 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:29.169869 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:29.169873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:29.169896 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:29.170214 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:29.170223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.171300 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.171307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:29.172899 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:29.172911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.174339 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.174346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:29.174560 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.174594 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 09:14:29.174719 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 09:14:29.174862 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":707,"threads_started":1}
I20260925 09:14:29.176308 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:29.167325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:34787 (local address 127.3.19.66:37839)
0925 09:14:29.167484 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:29.167488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:29.168666 (+ 1178us) server_negotiation.cc:407] Connection header received
0925 09:14:29.168941 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:29.168947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:29.169042 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:29.169107 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:29.169963 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.171100 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.173761 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.175028 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.175134 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.175230 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:29.175862 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 09:14:29.176007 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 09:14:29.176608 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d18be178e0c4916b60ac096ae5bebff" candidate_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" is_pre_election: true
I20260925 09:14:29.177060 11347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:29.167147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:37839 (local address 127.3.19.65:34787)
0925 09:14:29.168544 (+ 1397us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:29.168567 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:29.168780 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:29.169335 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:29.169339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:29.169365 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:29.169775 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:29.169789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.171291 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.171296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:29.173565 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:29.173580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.175440 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.175448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:29.175669 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.175703 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 09:14:29.176566 (+ 863us) client_negotiation.cc:240] Negotiation successful
0925 09:14:29.176729 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1239,"thread_start_us":1087,"threads_started":1}
I20260925 09:14:29.177086 11235 raft_consensus.cc:2454] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4f2a8bf0444ccb9e2e14bec877ddc9 in term 0.
I20260925 09:14:29.177814 10887 leader_election.cc:302] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee4f2a8bf0444ccb9e2e14bec877ddc9, f5fc6ff41f7a4628be2bf3ba26da9356; no voters:
I20260925 09:14:29.177881 11101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d18be178e0c4916b60ac096ae5bebff" candidate_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c539aaf8bdec43728f8e8870d05f91d6" is_pre_election: true
I20260925 09:14:29.178153 11101 raft_consensus.cc:2454] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4f2a8bf0444ccb9e2e14bec877ddc9 in term 0.
I20260925 09:14:29.178452 11327 raft_consensus.cc:2781] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:14:29.178577 11327 raft_consensus.cc:482] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:14:29.178692 11327 raft_consensus.cc:3037] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:29.181943 11327 raft_consensus.cc:504] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.182654 11327 leader_election.cc:288] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [CANDIDATE]: Term 1 election: Requested vote from peers c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66:37839), f5fc6ff41f7a4628be2bf3ba26da9356 (127.3.19.67:33599)
I20260925 09:14:29.183131 11101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d18be178e0c4916b60ac096ae5bebff" candidate_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c539aaf8bdec43728f8e8870d05f91d6"
I20260925 09:14:29.183229 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d18be178e0c4916b60ac096ae5bebff" candidate_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356"
I20260925 09:14:29.183359 11101 raft_consensus.cc:3037] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:29.183477 11235 raft_consensus.cc:3037] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:29.185876 11235 raft_consensus.cc:2454] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4f2a8bf0444ccb9e2e14bec877ddc9 in term 1.
I20260925 09:14:29.186305 10887 leader_election.cc:302] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee4f2a8bf0444ccb9e2e14bec877ddc9, f5fc6ff41f7a4628be2bf3ba26da9356; no voters:
I20260925 09:14:29.186376 11101 raft_consensus.cc:2454] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4f2a8bf0444ccb9e2e14bec877ddc9 in term 1.
I20260925 09:14:29.186719 11327 raft_consensus.cc:2781] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:29.187196 11327 raft_consensus.cc:686] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 LEADER]: Becoming Leader. State: Replica: ee4f2a8bf0444ccb9e2e14bec877ddc9, State: Running, Role: LEADER
I20260925 09:14:29.187645 11327 consensus_queue.cc:237] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.191448 10796 catalog_manager.cc:5495] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee4f2a8bf0444ccb9e2e14bec877ddc9 (127.3.19.65). New cstate: current_term: 1 leader_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } health_report { overall_health: UNKNOWN } } }
Sep 25 09:14:29 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327666, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.126@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:29.233580 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:29.219759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.3.19.126:37977)
0925 09:14:29.220710 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:29.220717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:29.220743 (+ 26us) server_negotiation.cc:407] Connection header received
0925 09:14:29.220781 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:29.220784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:29.220835 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:29.220948 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:29.222158 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.222901 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.224235 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.224628 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.224768 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.228248 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:29.228253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:29.228256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:29.228282 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:29.230879 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:29.231572 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:29.231578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:29.231582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:29.231705 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:29.232054 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:29.232061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:29.232064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:29.232317 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:29.232470 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:29.232858 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 09:14:29.233126 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":813,"thread_start_us":658,"threads_started":1}
I20260925 09:14:29.233659 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:29.219692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:37977 (local address 127.0.0.1:49268)
0925 09:14:29.220377 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:29.220410 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:29.220603 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:29.220971 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:29.220975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:29.221754 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:29.221978 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:29.221987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.223024 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.223029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:29.224103 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:29.224115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.224811 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.224817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:29.225016 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.225818 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:29.225834 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:29.228044 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:29.231019 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:29.231033 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:29.231041 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:29.231440 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:29.231808 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:29.231814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:29.231818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:29.231952 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:29.232446 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:29.232455 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:29.232623 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 09:14:29.232789 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 09:14:29.233264 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":170,"thread_start_us":363,"threads_started":1}
I20260925 09:14:29.239384 10796 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:49268:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 09:14:29.240479 10796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:14:29.247768 11079 tablet_service.cc:1467] Processing CreateTablet for tablet 4ccf52c27370485995c673779ad0d749 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5dd46b82306a40f3b2ccf01a58e6b58d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 09:14:29.247746 10934 tablet_service.cc:1467] Processing CreateTablet for tablet 4ccf52c27370485995c673779ad0d749 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5dd46b82306a40f3b2ccf01a58e6b58d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 09:14:29.248338 11079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ccf52c27370485995c673779ad0d749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:29.248338 10934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ccf52c27370485995c673779ad0d749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:29.253931 11215 tablet_service.cc:1467] Processing CreateTablet for tablet 4ccf52c27370485995c673779ad0d749 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5dd46b82306a40f3b2ccf01a58e6b58d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 09:14:29.254551 11215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ccf52c27370485995c673779ad0d749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:29.255959 11341 tablet_bootstrap.cc:492] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9: Bootstrap starting.
I20260925 09:14:29.256660 11343 tablet_bootstrap.cc:492] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6: Bootstrap starting.
I20260925 09:14:29.257663 11341 tablet_bootstrap.cc:654] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:29.258097 11343 tablet_bootstrap.cc:654] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:29.260371 11341 tablet_bootstrap.cc:492] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9: No bootstrap required, opened a new log
I20260925 09:14:29.260582 11341 ts_tablet_manager.cc:1397] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 09:14:29.261620 11341 raft_consensus.cc:348] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.261991 11341 raft_consensus.cc:374] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:29.262073 11341 raft_consensus.cc:729] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4f2a8bf0444ccb9e2e14bec877ddc9, State: Initialized, Role: FOLLOWER
I20260925 09:14:29.262394 11341 consensus_queue.cc:260] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.262939 11327 tablet_replica.cc:439] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } } }
I20260925 09:14:29.263144 11341 ts_tablet_manager.cc:1428] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 09:14:29.263247 11327 tablet_replica.cc:442] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:14:29.266237 11344 tablet_bootstrap.cc:492] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356: Bootstrap starting.
I20260925 09:14:29.267786 11343 tablet_bootstrap.cc:492] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6: No bootstrap required, opened a new log
I20260925 09:14:29.267985 11343 ts_tablet_manager.cc:1397] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260925 09:14:29.268194 11344 tablet_bootstrap.cc:654] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:29.268676 11343 raft_consensus.cc:348] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.268916 11343 raft_consensus.cc:374] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:29.269018 11343 raft_consensus.cc:729] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c539aaf8bdec43728f8e8870d05f91d6, State: Initialized, Role: FOLLOWER
I20260925 09:14:29.269382 11343 consensus_queue.cc:260] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.269886 11328 tablet_replica.cc:439] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } } }
I20260925 09:14:29.270267 11328 tablet_replica.cc:442] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:14:29.270704 11343 ts_tablet_manager.cc:1428] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 09:14:29.271624 11344 tablet_bootstrap.cc:492] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356: No bootstrap required, opened a new log
I20260925 09:14:29.271819 11344 ts_tablet_manager.cc:1397] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260925 09:14:29.272511 11344 raft_consensus.cc:348] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.272809 11344 raft_consensus.cc:374] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:29.272907 11344 raft_consensus.cc:729] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fc6ff41f7a4628be2bf3ba26da9356, State: Initialized, Role: FOLLOWER
I20260925 09:14:29.273152 11344 consensus_queue.cc:260] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.273648 11333 tablet_replica.cc:439] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } } }
I20260925 09:14:29.274042 11333 tablet_replica.cc:442] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:14:29.274984 11344 ts_tablet_manager.cc:1428] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 09:14:29.396060 11327 raft_consensus.cc:482] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:14:29.396384 11327 raft_consensus.cc:504] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.397387 11327 leader_election.cc:288] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66:37839), f5fc6ff41f7a4628be2bf3ba26da9356 (127.3.19.67:33599)
I20260925 09:14:29.398123 11101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ccf52c27370485995c673779ad0d749" candidate_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c539aaf8bdec43728f8e8870d05f91d6" is_pre_election: true
I20260925 09:14:29.398164 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ccf52c27370485995c673779ad0d749" candidate_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" is_pre_election: true
I20260925 09:14:29.398458 11101 raft_consensus.cc:2454] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4f2a8bf0444ccb9e2e14bec877ddc9 in term 0.
I20260925 09:14:29.398479 11235 raft_consensus.cc:2454] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4f2a8bf0444ccb9e2e14bec877ddc9 in term 0.
I20260925 09:14:29.399152 10887 leader_election.cc:302] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee4f2a8bf0444ccb9e2e14bec877ddc9, f5fc6ff41f7a4628be2bf3ba26da9356; no voters:
I20260925 09:14:29.399556 11327 raft_consensus.cc:2781] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:14:29.399678 11327 raft_consensus.cc:482] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:14:29.399802 11327 raft_consensus.cc:3037] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:29.402531 11327 raft_consensus.cc:504] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.403301 11327 leader_election.cc:288] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [CANDIDATE]: Term 1 election: Requested vote from peers c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66:37839), f5fc6ff41f7a4628be2bf3ba26da9356 (127.3.19.67:33599)
I20260925 09:14:29.404083 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ccf52c27370485995c673779ad0d749" candidate_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356"
I20260925 09:14:29.404260 11101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ccf52c27370485995c673779ad0d749" candidate_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c539aaf8bdec43728f8e8870d05f91d6"
I20260925 09:14:29.404347 11235 raft_consensus.cc:3037] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:29.404507 11101 raft_consensus.cc:3037] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:29.407787 11101 raft_consensus.cc:2454] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4f2a8bf0444ccb9e2e14bec877ddc9 in term 1.
I20260925 09:14:29.407807 11235 raft_consensus.cc:2454] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4f2a8bf0444ccb9e2e14bec877ddc9 in term 1.
I20260925 09:14:29.408384 10887 leader_election.cc:302] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee4f2a8bf0444ccb9e2e14bec877ddc9, f5fc6ff41f7a4628be2bf3ba26da9356; no voters:
I20260925 09:14:29.408727 11327 raft_consensus.cc:2781] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:29.408963 11327 raft_consensus.cc:686] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 LEADER]: Becoming Leader. State: Replica: ee4f2a8bf0444ccb9e2e14bec877ddc9, State: Running, Role: LEADER
I20260925 09:14:29.409304 11327 consensus_queue.cc:237] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.411073 11349 tablet_replica.cc:439] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9: TxnStatusTablet state changed. Reason: New leader ee4f2a8bf0444ccb9e2e14bec877ddc9. Latest consensus state: current_term: 1 leader_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } } }
I20260925 09:14:29.411419 11349 tablet_replica.cc:442] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9: This TxnStatusTablet replica's current role is: LEADER
I20260925 09:14:29.412578 11362 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 09:14:29.412827 11362 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 09:14:29.413288 10796 catalog_manager.cc:5495] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee4f2a8bf0444ccb9e2e14bec877ddc9 (127.3.19.65). New cstate: current_term: 1 leader_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } health_report { overall_health: UNKNOWN } } }
I20260925 09:14:29.434963 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:29.426592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:33377 (local address 127.0.0.1:42260)
0925 09:14:29.426971 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:29.426992 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:29.427139 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:29.428188 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:29.428195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:29.428209 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 09:14:29.428412 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:29.428424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.430043 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.430048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:29.431677 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:29.431695 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.432719 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.432725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:29.432940 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.433044 (+ 104us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 09:14:29.434320 (+ 1276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 09:14:29.434346 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 09:14:29.434444 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 09:14:29.434615 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260925 09:14:29.434980 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:29.426756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.3.19.65:33377)
0925 09:14:29.427729 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:29.427736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:29.427756 (+ 20us) server_negotiation.cc:407] Connection header received
0925 09:14:29.427860 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:29.427865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:29.427963 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:29.428144 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:14:29.428620 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.429882 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.431850 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.432529 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.432683 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.433364 (+ 681us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:14:29.434182 (+ 818us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:14:29.434342 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:29.434493 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 09:14:29.434587 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":676,"threads_started":1}
I20260925 09:14:29.440099 11101 raft_consensus.cc:1261] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Refusing update from remote peer ee4f2a8bf0444ccb9e2e14bec877ddc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:29.441036 11235 raft_consensus.cc:1261] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Refusing update from remote peer ee4f2a8bf0444ccb9e2e14bec877ddc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:29.441577 11327 consensus_queue.cc:1035] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:29.442216 11327 consensus_queue.cc:1035] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:29.445959 11328 tablet_replica.cc:439] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6: TxnStatusTablet state changed. Reason: New leader ee4f2a8bf0444ccb9e2e14bec877ddc9. Latest consensus state: current_term: 1 leader_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } } }
I20260925 09:14:29.446295 11328 tablet_replica.cc:442] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:14:29.448853 11349 tablet_replica.cc:439] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } } }
I20260925 09:14:29.449129 11349 tablet_replica.cc:442] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9: This TxnStatusTablet replica's current role is: LEADER
I20260925 09:14:29.450167 11330 mvcc.cc:205] Tried to move back new op lower bound from 7333182134017896448 to 7333182133903466496. Current Snapshot: MvccSnapshot[applied={T|T < 7333182134017896448 or (T in {7333182134017896448})}]
I20260925 09:14:29.455250 11328 tablet_replica.cc:439] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } } }
I20260925 09:14:29.455515 11328 tablet_replica.cc:442] T 4ccf52c27370485995c673779ad0d749 P c539aaf8bdec43728f8e8870d05f91d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:14:29.456068 11329 mvcc.cc:205] Tried to move back new op lower bound from 7333182134017896448 to 7333182133903466496. Current Snapshot: MvccSnapshot[applied={T|T < 7333182134017896448 or (T in {7333182134017896448})}]
I20260925 09:14:29.466414 11333 tablet_replica.cc:439] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356: TxnStatusTablet state changed. Reason: New leader ee4f2a8bf0444ccb9e2e14bec877ddc9. Latest consensus state: current_term: 1 leader_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } } }
I20260925 09:14:29.466761 11333 tablet_replica.cc:442] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:14:29.467386 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:29.460049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42266 (local address 127.3.19.65:33377)
0925 09:14:29.460264 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:29.460271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:29.460291 (+ 20us) server_negotiation.cc:407] Connection header received
0925 09:14:29.460359 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:29.460364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:29.460446 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:29.460540 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:14:29.461117 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.462364 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.464451 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.465134 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.465295 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.465914 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:14:29.466052 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:14:29.466160 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:29.466977 (+ 817us) server_negotiation.cc:299] Negotiation successful
0925 09:14:29.467072 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 09:14:29.469589 11327 tablet_replica.cc:439] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } } }
I20260925 09:14:29.469878 11327 tablet_replica.cc:442] T 4ccf52c27370485995c673779ad0d749 P ee4f2a8bf0444ccb9e2e14bec877ddc9: This TxnStatusTablet replica's current role is: LEADER
I20260925 09:14:29.474781 11333 tablet_replica.cc:439] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } } }
I20260925 09:14:29.475140 11333 tablet_replica.cc:442] T 4ccf52c27370485995c673779ad0d749 P f5fc6ff41f7a4628be2bf3ba26da9356: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 09:14:29.493772 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:29.487833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:46293 (local address 127.3.19.65:33377)
0925 09:14:29.488046 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:29.488055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:29.488649 (+ 594us) server_negotiation.cc:407] Connection header received
0925 09:14:29.488868 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:29.488873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:29.488953 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:29.489050 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:14:29.489633 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.490395 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.491853 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.492300 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.492421 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.492898 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:14:29.492995 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:14:29.493092 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:29.493256 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 09:14:29.493333 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":47}
I20260925 09:14:29.493794 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:29.487807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:33377 (local address 127.3.19.65:46293)
0925 09:14:29.488544 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:29.488561 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:29.488734 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:29.489057 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:29.489061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:29.489071 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 09:14:29.489489 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:29.489498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.490517 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.490525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:29.491714 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:29.491725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.492487 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.492493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:29.492652 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.492714 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 09:14:29.493092 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 09:14:29.493111 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 09:14:29.493204 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 09:14:29.493354 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":22,"thread_start_us":447,"threads_started":1}
I20260925 09:14:29.506748 11101 raft_consensus.cc:1261] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Refusing update from remote peer ee4f2a8bf0444ccb9e2e14bec877ddc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:29.506752 11235 raft_consensus.cc:1261] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Refusing update from remote peer ee4f2a8bf0444ccb9e2e14bec877ddc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:29.507510 11349 consensus_queue.cc:1035] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:29.508076 11327 consensus_queue.cc:1035] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:29.607118 10796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:29.607371 10796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:29.610569 10796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807: Sending DeleteTablet for 3 replicas of tablet 3d18be178e0c4916b60ac096ae5bebff
I20260925 09:14:29.611498 11079 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d18be178e0c4916b60ac096ae5bebff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:29 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:34418
I20260925 09:14:29.611826 11338 tablet_replica.cc:330] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6: stopping tablet replica
I20260925 09:14:29.612062 11215 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d18be178e0c4916b60ac096ae5bebff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:29 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:54038
I20260925 09:14:29.612078 11338 raft_consensus.cc:2229] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:29.612298 11338 raft_consensus.cc:2258] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:29.612345 11339 tablet_replica.cc:330] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356: stopping tablet replica
I20260925 09:14:29.612547 11339 raft_consensus.cc:2229] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:29.612761 11339 raft_consensus.cc:2258] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:29.613652 11338 ts_tablet_manager.cc:1905] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:29.614068 11339 ts_tablet_manager.cc:1905] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:29.617458 10934 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d18be178e0c4916b60ac096ae5bebff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:29 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:51530
I20260925 09:14:29.620735 11338 ts_tablet_manager.cc:1918] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:14:29.620873 11338 log.cc:1196] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/3d18be178e0c4916b60ac096ae5bebff
I20260925 09:14:29.621318 11338 ts_tablet_manager.cc:1939] T 3d18be178e0c4916b60ac096ae5bebff P c539aaf8bdec43728f8e8870d05f91d6: Deleting consensus metadata
I20260925 09:14:29.621412 11339 ts_tablet_manager.cc:1918] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:14:29.621559 11339 log.cc:1196] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/3d18be178e0c4916b60ac096ae5bebff
I20260925 09:14:29.621958 11339 ts_tablet_manager.cc:1939] T 3d18be178e0c4916b60ac096ae5bebff P f5fc6ff41f7a4628be2bf3ba26da9356: Deleting consensus metadata
I20260925 09:14:29.622560 10784 catalog_manager.cc:4841] TS c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66:37839): tablet 3d18be178e0c4916b60ac096ae5bebff (table test-table [id=7da4f9493143432a9d2c583a8ab2a169]) successfully deleted
I20260925 09:14:29.622989 11340 tablet_replica.cc:330] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9: stopping tablet replica
I20260925 09:14:29.623219 11340 raft_consensus.cc:2229] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:14:29.623786 11340 raft_consensus.cc:2258] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:29.625329 11340 ts_tablet_manager.cc:1905] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:29.637198 10784 catalog_manager.cc:4841] TS f5fc6ff41f7a4628be2bf3ba26da9356 (127.3.19.67:33599): tablet 3d18be178e0c4916b60ac096ae5bebff (table test-table [id=7da4f9493143432a9d2c583a8ab2a169]) successfully deleted
I20260925 09:14:29.640247 11340 ts_tablet_manager.cc:1918] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:14:29.643368 11340 log.cc:1196] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/3d18be178e0c4916b60ac096ae5bebff
I20260925 09:14:29.643987 11340 ts_tablet_manager.cc:1939] T 3d18be178e0c4916b60ac096ae5bebff P ee4f2a8bf0444ccb9e2e14bec877ddc9: Deleting consensus metadata
I20260925 09:14:29.645294 10784 catalog_manager.cc:4841] TS ee4f2a8bf0444ccb9e2e14bec877ddc9 (127.3.19.65:33377): tablet 3d18be178e0c4916b60ac096ae5bebff (table test-table [id=7da4f9493143432a9d2c583a8ab2a169]) successfully deleted
Sep 25 09:14:29 dist-test-slave-l6qh krb5kdc[10752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 09:14:29 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:29 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 09:14:29.711040 11357 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:29.698989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.3.19.126:37977)
0925 09:14:29.699287 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:29.699294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:29.699646 (+ 352us) server_negotiation.cc:407] Connection header received
0925 09:14:29.699845 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:29.699849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:29.699914 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:29.700008 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:29.701820 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.702546 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.704487 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.704927 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.705078 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.710775 (+ 5697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60620: BlockingRecv error: recv got EOF from 127.0.0.1:60620 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 09:14:29 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM
I20260925 09:14:29.727753 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:29.713750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.3.19.126:37977)
0925 09:14:29.713930 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:29.713937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:29.714326 (+ 389us) server_negotiation.cc:407] Connection header received
0925 09:14:29.714487 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:29.714492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:29.714567 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:29.714671 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:29.715841 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.716558 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.717845 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:29.718259 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:29.718390 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:29.722384 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:29.722389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:29.722392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:29.722452 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:29.725191 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:29.725893 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:29.725899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:29.725902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:29.726045 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:29.726419 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:29.726425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:29.726428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:29.726743 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:29.727001 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:29.727308 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 09:14:29.727474 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 09:14:29.731618 10796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:14:29.732345 10796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:14:29.739331 10934 tablet_service.cc:1467] Processing CreateTablet for tablet 59e90679b5cd4b06804295e9546e67ef (DEFAULT_TABLE table=test-table [id=c2df0c92f6d243929cb36ca82a5434b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:29.739791 10934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e90679b5cd4b06804295e9546e67ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:29.740168 11215 tablet_service.cc:1467] Processing CreateTablet for tablet 59e90679b5cd4b06804295e9546e67ef (DEFAULT_TABLE table=test-table [id=c2df0c92f6d243929cb36ca82a5434b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:29.740168 11079 tablet_service.cc:1467] Processing CreateTablet for tablet 59e90679b5cd4b06804295e9546e67ef (DEFAULT_TABLE table=test-table [id=c2df0c92f6d243929cb36ca82a5434b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:29.740592 11215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e90679b5cd4b06804295e9546e67ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:29.740592 11079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e90679b5cd4b06804295e9546e67ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:29.746688 11341 tablet_bootstrap.cc:492] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9: Bootstrap starting.
I20260925 09:14:29.747819 11343 tablet_bootstrap.cc:492] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6: Bootstrap starting.
I20260925 09:14:29.747925 11344 tablet_bootstrap.cc:492] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356: Bootstrap starting.
I20260925 09:14:29.748373 11341 tablet_bootstrap.cc:654] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:29.749593 11344 tablet_bootstrap.cc:654] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:29.749593 11343 tablet_bootstrap.cc:654] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:29.760237 11344 tablet_bootstrap.cc:492] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356: No bootstrap required, opened a new log
I20260925 09:14:29.760437 11344 ts_tablet_manager.cc:1397] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.006s
I20260925 09:14:29.761152 11344 raft_consensus.cc:348] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.761536 11344 raft_consensus.cc:374] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:29.761638 11344 raft_consensus.cc:729] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fc6ff41f7a4628be2bf3ba26da9356, State: Initialized, Role: FOLLOWER
I20260925 09:14:29.761914 11344 consensus_queue.cc:260] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.764948 11344 ts_tablet_manager.cc:1428] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 09:14:29.765661 11341 tablet_bootstrap.cc:492] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9: No bootstrap required, opened a new log
I20260925 09:14:29.765829 11341 ts_tablet_manager.cc:1397] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.015s
I20260925 09:14:29.766271 11343 tablet_bootstrap.cc:492] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6: No bootstrap required, opened a new log
I20260925 09:14:29.766499 11343 ts_tablet_manager.cc:1397] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.007s
I20260925 09:14:29.766707 11341 raft_consensus.cc:348] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.766984 11341 raft_consensus.cc:374] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:29.767095 11341 raft_consensus.cc:729] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4f2a8bf0444ccb9e2e14bec877ddc9, State: Initialized, Role: FOLLOWER
I20260925 09:14:29.767359 11341 consensus_queue.cc:260] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.768186 11343 raft_consensus.cc:348] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.768519 11343 raft_consensus.cc:374] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:29.768671 11343 raft_consensus.cc:729] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c539aaf8bdec43728f8e8870d05f91d6, State: Initialized, Role: FOLLOWER
I20260925 09:14:29.768699 11341 ts_tablet_manager.cc:1428] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 09:14:29.768998 11343 consensus_queue.cc:260] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.769647 11343 ts_tablet_manager.cc:1428] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 09:14:29.984138 11333 raft_consensus.cc:482] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:14:29.984377 11333 raft_consensus.cc:504] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.985090 11333 leader_election.cc:288] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee4f2a8bf0444ccb9e2e14bec877ddc9 (127.3.19.65:33377), c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66:37839)
I20260925 09:14:29.985829 11100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e90679b5cd4b06804295e9546e67ef" candidate_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c539aaf8bdec43728f8e8870d05f91d6" is_pre_election: true
I20260925 09:14:29.985780 10960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e90679b5cd4b06804295e9546e67ef" candidate_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" is_pre_election: true
I20260925 09:14:29.986083 10960 raft_consensus.cc:2454] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5fc6ff41f7a4628be2bf3ba26da9356 in term 0.
I20260925 09:14:29.986078 11100 raft_consensus.cc:2454] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5fc6ff41f7a4628be2bf3ba26da9356 in term 0.
I20260925 09:14:29.986558 11169 leader_election.cc:302] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee4f2a8bf0444ccb9e2e14bec877ddc9, f5fc6ff41f7a4628be2bf3ba26da9356; no voters:
I20260925 09:14:29.986816 11333 raft_consensus.cc:2781] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:14:29.986946 11333 raft_consensus.cc:482] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:14:29.987035 11333 raft_consensus.cc:3037] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:29.990032 11333 raft_consensus.cc:504] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.990679 11333 leader_election.cc:288] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [CANDIDATE]: Term 1 election: Requested vote from peers ee4f2a8bf0444ccb9e2e14bec877ddc9 (127.3.19.65:33377), c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66:37839)
I20260925 09:14:29.991385 10960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e90679b5cd4b06804295e9546e67ef" candidate_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9"
I20260925 09:14:29.991470 11100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e90679b5cd4b06804295e9546e67ef" candidate_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c539aaf8bdec43728f8e8870d05f91d6"
I20260925 09:14:29.991645 10960 raft_consensus.cc:3037] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:29.991699 11100 raft_consensus.cc:3037] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:29.994657 10960 raft_consensus.cc:2454] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5fc6ff41f7a4628be2bf3ba26da9356 in term 1.
I20260925 09:14:29.995127 11169 leader_election.cc:302] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee4f2a8bf0444ccb9e2e14bec877ddc9, f5fc6ff41f7a4628be2bf3ba26da9356; no voters:
I20260925 09:14:29.995191 11100 raft_consensus.cc:2454] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5fc6ff41f7a4628be2bf3ba26da9356 in term 1.
I20260925 09:14:29.995482 11333 raft_consensus.cc:2781] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:29.995893 11333 raft_consensus.cc:686] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 LEADER]: Becoming Leader. State: Replica: f5fc6ff41f7a4628be2bf3ba26da9356, State: Running, Role: LEADER
I20260925 09:14:29.996201 11333 consensus_queue.cc:237] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:29.999017 10796 catalog_manager.cc:5495] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5fc6ff41f7a4628be2bf3ba26da9356 (127.3.19.67). New cstate: current_term: 1 leader_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } health_report { overall_health: HEALTHY } } }
I20260925 09:14:30.067034 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:30.057279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51198 (local address 127.3.19.67:33599)
0925 09:14:30.061713 (+ 4434us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:30.061720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:30.061742 (+ 22us) server_negotiation.cc:407] Connection header received
0925 09:14:30.061817 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:30.061822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:30.061911 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:30.062038 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:14:30.062425 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.063587 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.064920 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.065631 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.065765 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:30.066209 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:14:30.066342 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:14:30.066434 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:30.066565 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 09:14:30.066648 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4295,"thread_start_us":4139,"threads_started":1}
I20260925 09:14:30.069877 11100 raft_consensus.cc:1261] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Refusing update from remote peer f5fc6ff41f7a4628be2bf3ba26da9356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:30.070072 10960 raft_consensus.cc:1261] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Refusing update from remote peer f5fc6ff41f7a4628be2bf3ba26da9356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:30.070772 11391 consensus_queue.cc:1035] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:30.071118 11333 consensus_queue.cc:1035] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:30.079361 11326 mvcc.cc:205] Tried to move back new op lower bound from 7333182136600002560 to 7333182136306954240. Current Snapshot: MvccSnapshot[applied={T|T < 7333182136600002560 or (T in {7333182136600002560})}]
I20260925 09:14:30.083766 10797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:30.084079 10797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:30.087426 10797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807: Sending DeleteTablet for 3 replicas of tablet 59e90679b5cd4b06804295e9546e67ef
I20260925 09:14:30.088303 11079 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e90679b5cd4b06804295e9546e67ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:30 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:34418
I20260925 09:14:30.088393 10934 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e90679b5cd4b06804295e9546e67ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:30 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:51530
I20260925 09:14:30.088596 11338 tablet_replica.cc:330] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6: stopping tablet replica
I20260925 09:14:30.088670 11340 tablet_replica.cc:330] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9: stopping tablet replica
I20260925 09:14:30.088796 11338 raft_consensus.cc:2229] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:30.088835 11340 raft_consensus.cc:2229] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:30.089061 11338 raft_consensus.cc:2258] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:30.089061 11340 raft_consensus.cc:2258] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:30.089289 11215 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e90679b5cd4b06804295e9546e67ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:30 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:54038
I20260925 09:14:30.089743 11339 tablet_replica.cc:330] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356: stopping tablet replica
I20260925 09:14:30.089921 11339 raft_consensus.cc:2229] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:14:30.089649 10797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 09:14:30.090130 11340 ts_tablet_manager.cc:1905] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:30.090269 11339 raft_consensus.cc:2258] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 09:14:30.090626 10797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:14:30.091297 11339 ts_tablet_manager.cc:1905] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:30.097486 11340 ts_tablet_manager.cc:1918] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:30.097645 11340 log.cc:1196] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/59e90679b5cd4b06804295e9546e67ef
I20260925 09:14:30.097724 11339 ts_tablet_manager.cc:1918] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:30.097857 11339 log.cc:1196] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/59e90679b5cd4b06804295e9546e67ef
I20260925 09:14:30.098058 11340 ts_tablet_manager.cc:1939] T 59e90679b5cd4b06804295e9546e67ef P ee4f2a8bf0444ccb9e2e14bec877ddc9: Deleting consensus metadata
I20260925 09:14:30.098368 11339 ts_tablet_manager.cc:1939] T 59e90679b5cd4b06804295e9546e67ef P f5fc6ff41f7a4628be2bf3ba26da9356: Deleting consensus metadata
I20260925 09:14:30.099308 11079 tablet_service.cc:1467] Processing CreateTablet for tablet a107059844d140c5b9191aff1b1f813e (DEFAULT_TABLE table=test-table [id=b773f92e873f49b4ba773fdaf8a161e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:30.099412 10784 catalog_manager.cc:4841] TS f5fc6ff41f7a4628be2bf3ba26da9356 (127.3.19.67:33599): tablet 59e90679b5cd4b06804295e9546e67ef (table test-table [id=c2df0c92f6d243929cb36ca82a5434b3]) successfully deleted
I20260925 09:14:30.099792 11079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a107059844d140c5b9191aff1b1f813e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:30.100452 11215 tablet_service.cc:1467] Processing CreateTablet for tablet a107059844d140c5b9191aff1b1f813e (DEFAULT_TABLE table=test-table [id=b773f92e873f49b4ba773fdaf8a161e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:30.100816 11215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a107059844d140c5b9191aff1b1f813e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:30.105302 10934 tablet_service.cc:1467] Processing CreateTablet for tablet a107059844d140c5b9191aff1b1f813e (DEFAULT_TABLE table=test-table [id=b773f92e873f49b4ba773fdaf8a161e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:14:30.106021 10934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a107059844d140c5b9191aff1b1f813e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:30.108274 11344 tablet_bootstrap.cc:492] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356: Bootstrap starting.
I20260925 09:14:30.110152 10784 catalog_manager.cc:4841] TS ee4f2a8bf0444ccb9e2e14bec877ddc9 (127.3.19.65:33377): tablet 59e90679b5cd4b06804295e9546e67ef (table test-table [id=c2df0c92f6d243929cb36ca82a5434b3]) successfully deleted
I20260925 09:14:30.110491 11344 tablet_bootstrap.cc:654] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:30.116181 11343 tablet_bootstrap.cc:492] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6: Bootstrap starting.
I20260925 09:14:30.117916 11343 tablet_bootstrap.cc:654] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:30.120028 11343 tablet_bootstrap.cc:492] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6: No bootstrap required, opened a new log
I20260925 09:14:30.120201 11343 ts_tablet_manager.cc:1397] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 09:14:30.120967 11343 raft_consensus.cc:348] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:30.121255 11343 raft_consensus.cc:374] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:30.121335 11343 raft_consensus.cc:729] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c539aaf8bdec43728f8e8870d05f91d6, State: Initialized, Role: FOLLOWER
I20260925 09:14:30.121677 11341 tablet_bootstrap.cc:492] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9: Bootstrap starting.
I20260925 09:14:30.121572 11343 consensus_queue.cc:260] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:30.122124 11343 ts_tablet_manager.cc:1428] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 09:14:30.124125 11341 tablet_bootstrap.cc:654] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:30.126212 11344 tablet_bootstrap.cc:492] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356: No bootstrap required, opened a new log
I20260925 09:14:30.126394 11344 ts_tablet_manager.cc:1397] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20260925 09:14:30.127177 11344 raft_consensus.cc:348] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:30.127472 11344 raft_consensus.cc:374] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:30.127545 11344 raft_consensus.cc:729] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fc6ff41f7a4628be2bf3ba26da9356, State: Initialized, Role: FOLLOWER
I20260925 09:14:30.127828 11344 consensus_queue.cc:260] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:30.128794 11344 ts_tablet_manager.cc:1428] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 09:14:30.134678 11338 ts_tablet_manager.cc:1905] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:30.144011 11338 ts_tablet_manager.cc:1918] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:14:30.144214 11338 log.cc:1196] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/59e90679b5cd4b06804295e9546e67ef
I20260925 09:14:30.144697 11338 ts_tablet_manager.cc:1939] T 59e90679b5cd4b06804295e9546e67ef P c539aaf8bdec43728f8e8870d05f91d6: Deleting consensus metadata
I20260925 09:14:30.147044 10784 catalog_manager.cc:4841] TS c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66:37839): tablet 59e90679b5cd4b06804295e9546e67ef (table test-table [id=c2df0c92f6d243929cb36ca82a5434b3]) successfully deleted
I20260925 09:14:30.147342 11341 tablet_bootstrap.cc:492] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9: No bootstrap required, opened a new log
I20260925 09:14:30.147579 11341 ts_tablet_manager.cc:1397] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.005s
I20260925 09:14:30.148350 11341 raft_consensus.cc:348] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:30.148674 11341 raft_consensus.cc:374] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:30.148761 11341 raft_consensus.cc:729] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4f2a8bf0444ccb9e2e14bec877ddc9, State: Initialized, Role: FOLLOWER
I20260925 09:14:30.149037 11341 consensus_queue.cc:260] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:30.149755 11341 ts_tablet_manager.cc:1428] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 09:14:30.234751 11328 raft_consensus.cc:482] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:14:30.235018 11328 raft_consensus.cc:504] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:30.236994 11328 leader_election.cc:288] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee4f2a8bf0444ccb9e2e14bec877ddc9 (127.3.19.65:33377), f5fc6ff41f7a4628be2bf3ba26da9356 (127.3.19.67:33599)
I20260925 09:14:30.249233 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:30.242141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:54143 (local address 127.3.19.67:33599)
0925 09:14:30.242349 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:30.242358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:30.243055 (+ 697us) server_negotiation.cc:407] Connection header received
0925 09:14:30.243305 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:30.243311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:30.243384 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:30.243475 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:30.244099 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.245179 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.247424 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.248183 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.248337 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:30.248455 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:30.248806 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 09:14:30.248950 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 09:14:30.249495 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:30.237184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:33377 (local address 127.3.19.66:58827)
0925 09:14:30.241727 (+ 4543us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:30.241754 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:30.241947 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:30.242965 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:30.242972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:30.243007 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:30.243319 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:30.243328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.244768 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.244775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:30.246846 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:30.246860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.248705 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.248714 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:30.248878 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:30.248908 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 09:14:30.248991 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 09:14:30.249139 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4395,"thread_start_us":4228,"threads_started":1}
I20260925 09:14:30.249970 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:30.242157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:33599 (local address 127.3.19.66:54143)
0925 09:14:30.242951 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:30.242971 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:30.243140 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:30.243503 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:30.243510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:30.243541 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:14:30.243902 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:30.243913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.245373 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.245378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:30.247277 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:30.247286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.248365 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.248371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:30.248595 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:30.248643 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 09:14:30.249547 (+ 904us) client_negotiation.cc:240] Negotiation successful
0925 09:14:30.249690 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":482,"threads_started":1}
I20260925 09:14:30.250022 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:30.237269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:58827 (local address 127.3.19.65:33377)
0925 09:14:30.242542 (+ 5273us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:30.242549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:30.242571 (+ 22us) server_negotiation.cc:407] Connection header received
0925 09:14:30.242646 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:30.242651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:30.242743 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:30.242902 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:14:30.243465 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.244629 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.247059 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.248425 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.248628 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:30.248764 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:30.249033 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 09:14:30.249184 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5092,"thread_start_us":5113,"threads_started":1}
I20260925 09:14:30.250634 10960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a107059844d140c5b9191aff1b1f813e" candidate_uuid: "c539aaf8bdec43728f8e8870d05f91d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" is_pre_election: true
I20260925 09:14:30.250630 11234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a107059844d140c5b9191aff1b1f813e" candidate_uuid: "c539aaf8bdec43728f8e8870d05f91d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" is_pre_election: true
I20260925 09:14:30.250892 11234 raft_consensus.cc:2454] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c539aaf8bdec43728f8e8870d05f91d6 in term 0.
I20260925 09:14:30.250896 10960 raft_consensus.cc:2454] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c539aaf8bdec43728f8e8870d05f91d6 in term 0.
I20260925 09:14:30.251415 11029 leader_election.cc:302] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c539aaf8bdec43728f8e8870d05f91d6, f5fc6ff41f7a4628be2bf3ba26da9356; no voters:
I20260925 09:14:30.251864 11328 raft_consensus.cc:2781] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:14:30.251997 11328 raft_consensus.cc:482] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:14:30.252141 11328 raft_consensus.cc:3037] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:30.254917 11328 raft_consensus.cc:504] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:30.255635 11328 leader_election.cc:288] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [CANDIDATE]: Term 1 election: Requested vote from peers ee4f2a8bf0444ccb9e2e14bec877ddc9 (127.3.19.65:33377), f5fc6ff41f7a4628be2bf3ba26da9356 (127.3.19.67:33599)
I20260925 09:14:30.256115 10960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a107059844d140c5b9191aff1b1f813e" candidate_uuid: "c539aaf8bdec43728f8e8870d05f91d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9"
I20260925 09:14:30.256413 10960 raft_consensus.cc:3037] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:30.256605 11234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a107059844d140c5b9191aff1b1f813e" candidate_uuid: "c539aaf8bdec43728f8e8870d05f91d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356"
I20260925 09:14:30.256841 11234 raft_consensus.cc:3037] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:30.259784 10960 raft_consensus.cc:2454] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c539aaf8bdec43728f8e8870d05f91d6 in term 1.
I20260925 09:14:30.259811 11234 raft_consensus.cc:2454] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c539aaf8bdec43728f8e8870d05f91d6 in term 1.
I20260925 09:14:30.260246 11029 leader_election.cc:302] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c539aaf8bdec43728f8e8870d05f91d6, ee4f2a8bf0444ccb9e2e14bec877ddc9; no voters:
I20260925 09:14:30.260567 11328 raft_consensus.cc:2781] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:30.261072 11328 raft_consensus.cc:686] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [term 1 LEADER]: Becoming Leader. State: Replica: c539aaf8bdec43728f8e8870d05f91d6, State: Running, Role: LEADER
I20260925 09:14:30.261480 11328 consensus_queue.cc:237] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } }
I20260925 09:14:30.265364 10797 catalog_manager.cc:5495] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66). New cstate: current_term: 1 leader_uuid: "c539aaf8bdec43728f8e8870d05f91d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c539aaf8bdec43728f8e8870d05f91d6" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 } health_report { overall_health: UNKNOWN } } }
I20260925 09:14:30.305419 11407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:30.299198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.3.19.66:37839)
0925 09:14:30.299794 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:30.299799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:30.299816 (+ 17us) server_negotiation.cc:407] Connection header received
0925 09:14:30.299892 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:30.299895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:30.299967 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:30.300151 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:14:30.300592 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.301524 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.302870 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.303308 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.303442 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:30.303954 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:14:30.304680 (+ 726us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:14:30.304848 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:30.304915 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 09:14:30.304997 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":346,"threads_started":1}
I20260925 09:14:30.324774 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:30.317478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:47371 (local address 127.3.19.65:33377)
0925 09:14:30.317696 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:30.317704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:30.318348 (+ 644us) server_negotiation.cc:407] Connection header received
0925 09:14:30.318565 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:30.318571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:30.318657 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:30.318755 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:14:30.319351 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.320528 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.322551 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.323273 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.323417 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:30.323946 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:14:30.324106 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:14:30.324198 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:30.324371 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 09:14:30.324454 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 09:14:30.324913 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:30.317375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:33377 (local address 127.3.19.66:47371)
0925 09:14:30.318203 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:30.318243 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:30.318417 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:30.318790 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:30.318795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:30.318809 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 09:14:30.319189 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:30.319202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.320679 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.320686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:30.322392 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:30.322408 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.323483 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.323490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:30.323661 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:30.323746 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 09:14:30.324224 (+ 478us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 09:14:30.324236 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 09:14:30.324334 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 09:14:30.324482 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":465,"threads_started":1}
I20260925 09:14:30.334210 10960 raft_consensus.cc:1261] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Refusing update from remote peer c539aaf8bdec43728f8e8870d05f91d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:30.334939 11234 raft_consensus.cc:1261] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Refusing update from remote peer c539aaf8bdec43728f8e8870d05f91d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:14:30.334944 11403 consensus_queue.cc:1035] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4f2a8bf0444ccb9e2e14bec877ddc9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:30.335546 11328 consensus_queue.cc:1035] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fc6ff41f7a4628be2bf3ba26da9356" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 33599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:14:30.384305 11407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:30.364697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:41281 (local address 127.3.19.66:37839)
0925 09:14:30.364848 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:30.364853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:30.377698 (+ 12845us) server_negotiation.cc:407] Connection header received
0925 09:14:30.377789 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:30.377796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:30.377886 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:30.377977 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:14:30.378753 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.379848 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.382105 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.382782 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.382903 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:30.383498 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:14:30.383647 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:14:30.383705 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:30.383913 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 09:14:30.383996 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 09:14:30.385066 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:30.364605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:37839 (local address 127.3.19.65:41281)
0925 09:14:30.374745 (+ 10140us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:30.374776 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:30.374927 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:30.378151 (+ 3224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:30.378156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:30.378170 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 09:14:30.378546 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:30.378560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.380018 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.380024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:30.381922 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:30.381950 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.383079 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.383086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:30.383269 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:30.383320 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 09:14:30.383786 (+ 466us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 09:14:30.383796 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 09:14:30.384515 (+ 719us) client_negotiation.cc:240] Negotiation successful
0925 09:14:30.384678 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9954,"thread_start_us":10423,"threads_started":1}
I20260925 09:14:30.437285 10797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:30.437511 10797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:14:30.440507 10797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 138ace9539dd42249048a32a0566d807: Sending DeleteTablet for 3 replicas of tablet a107059844d140c5b9191aff1b1f813e
I20260925 09:14:30.441398 10934 tablet_service.cc:1514] Processing DeleteTablet for tablet a107059844d140c5b9191aff1b1f813e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:30 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:51530
I20260925 09:14:30.441553 11079 tablet_service.cc:1514] Processing DeleteTablet for tablet a107059844d140c5b9191aff1b1f813e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:30 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:34418
I20260925 09:14:30.441741 11340 tablet_replica.cc:330] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9: stopping tablet replica
I20260925 09:14:30.441848 11215 tablet_service.cc:1514] Processing DeleteTablet for tablet a107059844d140c5b9191aff1b1f813e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:14:30 UTC) from {username='oryx', principal='oryx/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:54038
I20260925 09:14:30.441938 11340 raft_consensus.cc:2229] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:30.442123 11340 raft_consensus.cc:2258] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:30.442148 11338 tablet_replica.cc:330] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6: stopping tablet replica
I20260925 09:14:30.442343 11338 raft_consensus.cc:2229] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:14:30.442795 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 10868
I20260925 09:14:30.442880 11338 raft_consensus.cc:2258] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:30.443173 11339 tablet_replica.cc:330] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356: stopping tablet replica
I20260925 09:14:30.443375 11339 raft_consensus.cc:2229] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:14:30.443554 11339 raft_consensus.cc:2258] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:14:30.444979 11340 ts_tablet_manager.cc:1905] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:30.445572 11338 ts_tablet_manager.cc:1905] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:30.452720 11338 ts_tablet_manager.cc:1918] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:14:30.452879 11338 log.cc:1196] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/a107059844d140c5b9191aff1b1f813e
I20260925 09:14:30.453982 11338 ts_tablet_manager.cc:1939] T a107059844d140c5b9191aff1b1f813e P c539aaf8bdec43728f8e8870d05f91d6: Deleting consensus metadata
I20260925 09:14:30.454463 11339 ts_tablet_manager.cc:1905] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:14:30.456022 11340 ts_tablet_manager.cc:1918] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:14:30.456251 11340 log.cc:1196] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/a107059844d140c5b9191aff1b1f813e
I20260925 09:14:30.456739 11340 ts_tablet_manager.cc:1939] T a107059844d140c5b9191aff1b1f813e P ee4f2a8bf0444ccb9e2e14bec877ddc9: Deleting consensus metadata
I20260925 09:14:30.458046 10784 catalog_manager.cc:4841] TS c539aaf8bdec43728f8e8870d05f91d6 (127.3.19.66:37839): tablet a107059844d140c5b9191aff1b1f813e (table test-table [id=b773f92e873f49b4ba773fdaf8a161e5]) successfully deleted
I20260925 09:14:30.458366 10784 catalog_manager.cc:4841] TS ee4f2a8bf0444ccb9e2e14bec877ddc9 (127.3.19.65:33377): tablet a107059844d140c5b9191aff1b1f813e (table test-table [id=b773f92e873f49b4ba773fdaf8a161e5]) successfully deleted
I20260925 09:14:30.461477 11339 ts_tablet_manager.cc:1918] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 09:14:30.461659 11339 log.cc:1196] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/a107059844d140c5b9191aff1b1f813e
I20260925 09:14:30.462085 11339 ts_tablet_manager.cc:1939] T a107059844d140c5b9191aff1b1f813e P f5fc6ff41f7a4628be2bf3ba26da9356: Deleting consensus metadata
I20260925 09:14:30.462994 10784 catalog_manager.cc:4841] TS f5fc6ff41f7a4628be2bf3ba26da9356 (127.3.19.67:33599): tablet a107059844d140c5b9191aff1b1f813e (table test-table [id=b773f92e873f49b4ba773fdaf8a161e5]) successfully deleted
Sep 25 09:14:30 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.19.65@KRBTEST.COM
I20260925 09:14:30.472430 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:30.453417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42282 (local address 127.3.19.65:33377)
0925 09:14:30.453694 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:30.453704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:30.454447 (+ 743us) server_negotiation.cc:407] Connection header received
0925 09:14:30.454932 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:30.454938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:30.455029 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:30.455102 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:30.456912 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.458158 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.459971 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.460640 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.460789 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:30.465642 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:30.465663 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:30.465685 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:30.465729 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:30.469516 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:30.470192 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:30.470203 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:30.470207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:30.470358 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:30.470754 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:30.470760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:30.470763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:30.471226 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:30.471532 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:30.471778 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 09:14:30.472017 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 09:14:30.473176 11000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:30.630040 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 10868
I20260925 09:14:30.667516 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 11009
Sep 25 09:14:30 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.19.66@KRBTEST.COM
I20260925 09:14:30.687964 11407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:30.668302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37966 (local address 127.3.19.66:37839)
0925 09:14:30.668876 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:30.668883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:30.668899 (+ 16us) server_negotiation.cc:407] Connection header received
0925 09:14:30.668957 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:30.668962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:30.669053 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:30.669167 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:30.670879 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.672035 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.674129 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.674851 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.674970 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:30.680446 (+ 5476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:30.680460 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:30.680477 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:30.680523 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:30.684708 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:30.685705 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:30.685712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:30.685714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:30.685853 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:30.686324 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:30.686327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:30.686329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:30.686681 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:30.686932 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:30.687324 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 09:14:30.687504 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436}
I20260925 09:14:30.688820 11141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:30.811143 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 11009
I20260925 09:14:30.831688 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 11150
Sep 25 09:14:30 dist-test-slave-l6qh krb5kdc[10752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.19.67@KRBTEST.COM
I20260925 09:14:30.848110 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:30.832465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51206 (local address 127.3.19.67:33599)
0925 09:14:30.832889 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:30.832895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:30.832911 (+ 16us) server_negotiation.cc:407] Connection header received
0925 09:14:30.832963 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:30.832967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:30.833023 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:30.833113 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:30.834700 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.835371 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.836720 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:30.837112 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:30.837210 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:30.842210 (+ 5000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:30.842225 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:30.842241 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:30.842282 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:30.845412 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:30.846128 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:30.846135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:30.846137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:30.846272 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:30.846672 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:30.846675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:30.846677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:30.847031 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:30.847254 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:30.847565 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 09:14:30.847738 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":240,"threads_started":1}
I20260925 09:14:30.848826 11281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:30.970546 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 11150
I20260925 09:14:30.992453 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 10765
I20260925 09:14:30.993885 10837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:31.106400 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 10765
2026-09-25T09:14:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6257 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:14:31 dist-test-slave-l6qh krb5kdc[11427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:14:31 dist-test-slave-l6qh krb5kdc[11427](info): set up 2 sockets
Sep 25 09:14:31 dist-test-slave-l6qh krb5kdc[11427](info): commencing operation
krb5kdc: starting...
W20260925 09:14:32.227447 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:14:32 dist-test-slave-l6qh krb5kdc[11427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:14:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:14:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:14:32.469784 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:39881
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:39881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:32.807757 11440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:32.808061 11440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:32.808151 11440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:32.815445 11440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:32.815565 11440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:32.815613 11440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:32.815670 11440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:32.815711 11440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:14:32.825559 11440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:39881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:39881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:32.826854 11440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:32.828122 11440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:32.840232 11445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:32.840289 11448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:32.840201 11446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:32.841274 11440 server_base.cc:1034] running on GCE node
I20260925 09:14:32.842083 11440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:32.843597 11440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:32.844887 11440 hybrid_clock.cc:648] HybridClock initialized: now 1790327672844839 us; error 43 us; skew 500 ppm
I20260925 09:14:32.845355 11440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T09:14:32Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1811 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:14:33 dist-test-slave-l6qh krb5kdc[11454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:14:33 dist-test-slave-l6qh krb5kdc[11454](info): set up 2 sockets
Sep 25 09:14:33 dist-test-slave-l6qh krb5kdc[11454](info): commencing operation
krb5kdc: starting...
W20260925 09:14:34.035331 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:14:34 dist-test-slave-l6qh krb5kdc[11454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:14:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:14:34.286345 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:43785
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:43785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:34.627506 11467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:34.627822 11467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:34.627911 11467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:34.635250 11467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:34.635371 11467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:34.635418 11467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:34.635466 11467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:34.635511 11467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:14:34.645553 11467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:43785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:43785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:34.646878 11467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:34.648278 11467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:34.660909 11473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:34.660935 11475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:34.661017 11472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:34.661795 11467 server_base.cc:1034] running on GCE node
I20260925 09:14:34.662669 11467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:34.664177 11467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:34.665338 11467 hybrid_clock.cc:648] HybridClock initialized: now 1790327674665282 us; error 47 us; skew 500 ppm
I20260925 09:14:34.665663 11467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:34 dist-test-slave-l6qh krb5kdc[11454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327674, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:34.668872 11467 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:14:34.670908 11467 webserver.cc:460] Webserver started at http://127.3.19.126:35451/ using document root <none> and password file <none>
I20260925 09:14:34.671726 11467 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:34.671833 11467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:34.672106 11467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:34.675195 11467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "9d2357cc87144ccab54bab1b5238bc4f"
format_stamp: "Formatted at 2026-09-25 09:14:34 on dist-test-slave-l6qh"
server_key: "6477629d0f8dbb2ecaacf68b7e517510"
server_key_iv: "f3fd41b557a06f590eb8e17b3840925d"
server_key_version: "encryptionkey@0"
I20260925 09:14:34.675976 11467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "9d2357cc87144ccab54bab1b5238bc4f"
format_stamp: "Formatted at 2026-09-25 09:14:34 on dist-test-slave-l6qh"
server_key: "6477629d0f8dbb2ecaacf68b7e517510"
server_key_iv: "f3fd41b557a06f590eb8e17b3840925d"
server_key_version: "encryptionkey@0"
I20260925 09:14:34.682226 11467 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 09:14:34.686571 11482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:34.688380 11467 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260925 09:14:34.688597 11467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "9d2357cc87144ccab54bab1b5238bc4f"
format_stamp: "Formatted at 2026-09-25 09:14:34 on dist-test-slave-l6qh"
server_key: "6477629d0f8dbb2ecaacf68b7e517510"
server_key_iv: "f3fd41b557a06f590eb8e17b3840925d"
server_key_version: "encryptionkey@0"
I20260925 09:14:34.688848 11467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:34.712106 11467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:34.716576 11467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:34.716900 11467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:34.736563 11467 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:43785
I20260925 09:14:34.736603 11534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:43785 every 8 connection(s)
I20260925 09:14:34.738152 11467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:14:34.741365 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 11467
I20260925 09:14:34.741582 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:14:34.742203 11535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:34.742221 3149 external_mini_cluster.cc:1428] Setting key 4e5d48b725a79104e086dca1547b5f3a
I20260925 09:14:34.757212 11535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f: Bootstrap starting.
Sep 25 09:14:34 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:14:34.761111 11535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:34.762653 11535 log.cc:824] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:34.766531 11535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f: No bootstrap required, opened a new log
I20260925 09:14:34.768953 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:34.744323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.3.19.126:43785)
0925 09:14:34.745259 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:34.745275 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:34.745329 (+ 54us) server_negotiation.cc:407] Connection header received
0925 09:14:34.747417 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:34.747449 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:34.748068 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:34.748580 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:34.750069 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:34.752039 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:34.753717 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:34.754899 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:34.755124 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:34.759429 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:34.759445 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:34.759467 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:34.759517 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:34.763801 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:34.764762 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:34.764772 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:34.764784 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:34.765005 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:34.765485 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:34.765490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:34.765493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:34.766236 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:34.766557 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:34.766966 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 09:14:34.767476 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":359,"threads_started":1}
I20260925 09:14:34.776068 11535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2357cc87144ccab54bab1b5238bc4f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43785 } }
I20260925 09:14:34.776449 11535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:34.776526 11535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2357cc87144ccab54bab1b5238bc4f, State: Initialized, Role: FOLLOWER
I20260925 09:14:34.777034 11535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2357cc87144ccab54bab1b5238bc4f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43785 } }
I20260925 09:14:34.777354 11535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:14:34.777516 11535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:14:34.777755 11535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:34.780112 11535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2357cc87144ccab54bab1b5238bc4f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43785 } }
I20260925 09:14:34.780689 11535 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d2357cc87144ccab54bab1b5238bc4f; no voters:
I20260925 09:14:34.781425 11535 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:14:34.781607 11540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:34.782495 11540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 1 LEADER]: Becoming Leader. State: Replica: 9d2357cc87144ccab54bab1b5238bc4f, State: Running, Role: LEADER
I20260925 09:14:34.783102 11540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2357cc87144ccab54bab1b5238bc4f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43785 } }
I20260925 09:14:34.783610 11535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:34.786118 11542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2357cc87144ccab54bab1b5238bc4f. Latest consensus state: current_term: 1 leader_uuid: "9d2357cc87144ccab54bab1b5238bc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2357cc87144ccab54bab1b5238bc4f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43785 } } }
I20260925 09:14:34.786547 11542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:34.787171 11541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d2357cc87144ccab54bab1b5238bc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2357cc87144ccab54bab1b5238bc4f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43785 } } }
I20260925 09:14:34.787528 11541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:34.790380 11547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:14:34.796384 11547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:14:34.806763 11547 catalog_manager.cc:1347] Generated new cluster ID: 723efeae6a4846948553c30b32ca51a5
I20260925 09:14:34.806911 11547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:14:34.832500 11547 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:14:34.833981 11547 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:14:34.845052 11547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f: Generated new TSK 0
I20260925 09:14:34.845981 11547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:14:34.916234 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:43785
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:35.260206 11561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:35.260560 11561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:35.260748 11561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:35.268227 11561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:35.268392 11561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:35.268586 11561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:14:35.278924 11561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:43785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:35.280378 11561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:35.281921 11561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:35.296444 11566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:35.296512 11569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:35.296446 11567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:35.297382 11561 server_base.cc:1034] running on GCE node
I20260925 09:14:35.298040 11561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:35.299041 11561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:35.300252 11561 hybrid_clock.cc:648] HybridClock initialized: now 1790327675300215 us; error 49 us; skew 500 ppm
I20260925 09:14:35.300592 11561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:35 dist-test-slave-l6qh krb5kdc[11454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327675, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:35.303902 11561 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:14:35.305416 11561 webserver.cc:460] Webserver started at http://127.3.19.65:34749/ using document root <none> and password file <none>
I20260925 09:14:35.306090 11561 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:35.306190 11561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:35.306445 11561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:35.309415 11561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "d6f1cb51a5b14e39a48f480480284c21"
format_stamp: "Formatted at 2026-09-25 09:14:35 on dist-test-slave-l6qh"
server_key: "0df35554a7c982ed1ea46d64e491f87b"
server_key_iv: "b4725034b3a5ad70c95984d83c404ee6"
server_key_version: "encryptionkey@0"
I20260925 09:14:35.310185 11561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "d6f1cb51a5b14e39a48f480480284c21"
format_stamp: "Formatted at 2026-09-25 09:14:35 on dist-test-slave-l6qh"
server_key: "0df35554a7c982ed1ea46d64e491f87b"
server_key_iv: "b4725034b3a5ad70c95984d83c404ee6"
server_key_version: "encryptionkey@0"
I20260925 09:14:35.315392 11561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:14:35.318945 11576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:35.320374 11561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:14:35.320576 11561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "d6f1cb51a5b14e39a48f480480284c21"
format_stamp: "Formatted at 2026-09-25 09:14:35 on dist-test-slave-l6qh"
server_key: "0df35554a7c982ed1ea46d64e491f87b"
server_key_iv: "b4725034b3a5ad70c95984d83c404ee6"
server_key_version: "encryptionkey@0"
I20260925 09:14:35.320801 11561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:35.349148 11561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:35.353917 11561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:35.354250 11561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:35.355243 11561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:35.356601 11561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:35.356673 11561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:35.356813 11561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:35.356879 11561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:35.390910 11561 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:44893
I20260925 09:14:35.391076 11688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:44893 every 8 connection(s)
I20260925 09:14:35.392344 11561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:14:35.402841 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 11561
I20260925 09:14:35.403076 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:14:35.403741 3149 external_mini_cluster.cc:1428] Setting key 27d97f7e8de3a8c7348e474ecebbd251
Sep 25 09:14:35 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327675, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:14:35.414685 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:35.395119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:37147 (local address 127.3.19.126:43785)
0925 09:14:35.395563 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:35.395568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:35.396591 (+ 1023us) server_negotiation.cc:407] Connection header received
0925 09:14:35.398714 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:35.398720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:35.398823 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:35.398964 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:35.400630 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:35.401478 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:35.403245 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:35.403685 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:35.403842 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:35.407957 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:35.407966 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:35.407974 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:35.408027 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:35.411565 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:35.412350 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:35.412358 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:35.412364 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:35.412513 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:35.412930 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:35.412938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:35.412944 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:35.413367 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:35.413516 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:35.414241 (+ 725us) server_negotiation.cc:299] Negotiation successful
0925 09:14:35.414426 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":248,"threads_started":1}
I20260925 09:14:35.416239 11692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:35.395600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:37147)
0925 09:14:35.396382 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:35.396422 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:35.398396 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:35.399243 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:35.399252 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:35.399818 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:35.400425 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:35.400452 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:35.401675 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:35.401679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:35.403066 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:35.403079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:35.403940 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:35.403950 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:35.404229 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:35.405018 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:35.405049 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:35.407706 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:35.411733 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:35.411741 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:35.411755 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:35.412226 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:35.412630 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:35.412633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:35.412636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:35.412822 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:35.413483 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:35.413489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:35.413849 (+ 360us) client_negotiation.cc:769] Sending connection context
0925 09:14:35.414093 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 09:14:35.414732 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":309,"threads_started":1}
I20260925 09:14:35.417979 11689 heartbeater.cc:347] Connected to a master server at 127.3.19.126:43785
I20260925 09:14:35.418300 11689 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:35.419013 11689 heartbeater.cc:511] Master 127.3.19.126:43785 requested a full tablet report, sending...
I20260925 09:14:35.421201 11499 ts_manager.cc:194] Registered new tserver with Master: d6f1cb51a5b14e39a48f480480284c21 (127.3.19.65:44893)
I20260925 09:14:35.422715 11499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:37147
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:14:35.457458 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:43785
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:35.805300 11695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:35.805660 11695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:35.805837 11695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:35.813150 11695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:35.813371 11695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:35.813558 11695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:14:35.823709 11695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:43785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:35.825086 11695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:35.826426 11695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:35.840335 11701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:35.840332 11702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:35.840479 11704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:35.841159 11695 server_base.cc:1034] running on GCE node
I20260925 09:14:35.841784 11695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:35.842690 11695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:35.843894 11695 hybrid_clock.cc:648] HybridClock initialized: now 1790327675843825 us; error 62 us; skew 500 ppm
I20260925 09:14:35.844257 11695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:35 dist-test-slave-l6qh krb5kdc[11454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327675, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:35.847522 11695 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:14:35.849021 11695 webserver.cc:460] Webserver started at http://127.3.19.66:36181/ using document root <none> and password file <none>
I20260925 09:14:35.849767 11695 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:35.849907 11695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:35.850224 11695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:35.853287 11695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "71da1d8cf52d4bfdb824753a48ee38e6"
format_stamp: "Formatted at 2026-09-25 09:14:35 on dist-test-slave-l6qh"
server_key: "39209fa663f627601b256b6f3f91f044"
server_key_iv: "f410dc07f3acab26efd91eedf83145bf"
server_key_version: "encryptionkey@0"
I20260925 09:14:35.854096 11695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "71da1d8cf52d4bfdb824753a48ee38e6"
format_stamp: "Formatted at 2026-09-25 09:14:35 on dist-test-slave-l6qh"
server_key: "39209fa663f627601b256b6f3f91f044"
server_key_iv: "f410dc07f3acab26efd91eedf83145bf"
server_key_version: "encryptionkey@0"
I20260925 09:14:35.859367 11695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:14:35.862986 11711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:35.864450 11695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:14:35.864653 11695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "71da1d8cf52d4bfdb824753a48ee38e6"
format_stamp: "Formatted at 2026-09-25 09:14:35 on dist-test-slave-l6qh"
server_key: "39209fa663f627601b256b6f3f91f044"
server_key_iv: "f410dc07f3acab26efd91eedf83145bf"
server_key_version: "encryptionkey@0"
I20260925 09:14:35.864876 11695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:35.882651 11695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:35.887400 11695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:35.887709 11695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:35.888680 11695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:35.890208 11695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:35.890309 11695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:35.890453 11695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:35.890522 11695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:35.924551 11695 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:39443
I20260925 09:14:35.924600 11825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:39443 every 8 connection(s)
I20260925 09:14:35.925987 11695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:14:35.931705 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 11695
I20260925 09:14:35.931902 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:14:35.932497 3149 external_mini_cluster.cc:1428] Setting key 130ab58c49dc0d4a310f414515bbda6e
Sep 25 09:14:35 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327675, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:14:35.947362 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:35.928575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:59321 (local address 127.3.19.126:43785)
0925 09:14:35.929011 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:35.929015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:35.930065 (+ 1050us) server_negotiation.cc:407] Connection header received
0925 09:14:35.932173 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:35.932177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:35.932245 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:35.932344 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:35.934337 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:35.935008 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:35.936574 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:35.936994 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:35.937129 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:35.940927 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:35.940934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:35.940941 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:35.940982 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:35.943864 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:35.944849 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:35.944852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:35.944855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:35.944974 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:35.945521 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:35.945526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:35.945528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:35.945799 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:35.945927 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:35.946903 (+ 976us) server_negotiation.cc:299] Negotiation successful
0925 09:14:35.947106 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":202,"threads_started":1}
I20260925 09:14:35.948953 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:35.929160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:59321)
0925 09:14:35.929907 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:35.929949 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:35.931864 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:35.932638 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:35.932651 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:35.933450 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:35.934086 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:35.934130 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:35.935164 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:35.935171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:35.936422 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:35.936433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:35.937254 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:35.937263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:35.937499 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:35.938208 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:35.938236 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:35.940629 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:35.944017 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:35.944029 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:35.944051 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:35.944722 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:35.945092 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:35.945100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:35.945106 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:35.945383 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:35.945945 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:35.945957 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:35.946467 (+ 510us) client_negotiation.cc:769] Sending connection context
0925 09:14:35.946802 (+ 335us) client_negotiation.cc:240] Negotiation successful
0925 09:14:35.947403 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":264,"threads_started":1}
I20260925 09:14:35.950636 11826 heartbeater.cc:347] Connected to a master server at 127.3.19.126:43785
I20260925 09:14:35.951041 11826 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:35.951952 11826 heartbeater.cc:511] Master 127.3.19.126:43785 requested a full tablet report, sending...
I20260925 09:14:35.953856 11499 ts_manager.cc:194] Registered new tserver with Master: 71da1d8cf52d4bfdb824753a48ee38e6 (127.3.19.66:39443)
I20260925 09:14:35.954809 11499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:59321
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:14:35.988802 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:43785
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:36.327384 11832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:36.327715 11832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:36.327888 11832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:36.335269 11832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:36.335427 11832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:36.335613 11832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:14:36.345715 11832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:43785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:36.347067 11832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:36.348399 11832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:36.361820 11837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:36.361753 11838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:36.361783 11840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:36.362826 11832 server_base.cc:1034] running on GCE node
I20260925 09:14:36.363464 11832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:36.364454 11832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:36.365680 11832 hybrid_clock.cc:648] HybridClock initialized: now 1790327676365629 us; error 48 us; skew 500 ppm
I20260925 09:14:36.366005 11832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:36 dist-test-slave-l6qh krb5kdc[11454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327676, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:36.369573 11832 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:14:36.371230 11832 webserver.cc:460] Webserver started at http://127.3.19.67:43429/ using document root <none> and password file <none>
I20260925 09:14:36.372116 11832 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:36.372241 11832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:36.372545 11832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:36.376178 11832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "4c7ab7aa78c440208a3fdd68e7798cb1"
format_stamp: "Formatted at 2026-09-25 09:14:36 on dist-test-slave-l6qh"
server_key: "098345ab1bac65dd2b0675fed2e22e99"
server_key_iv: "1fa2170410b427565bc12399aa65b387"
server_key_version: "encryptionkey@0"
I20260925 09:14:36.377218 11832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "4c7ab7aa78c440208a3fdd68e7798cb1"
format_stamp: "Formatted at 2026-09-25 09:14:36 on dist-test-slave-l6qh"
server_key: "098345ab1bac65dd2b0675fed2e22e99"
server_key_iv: "1fa2170410b427565bc12399aa65b387"
server_key_version: "encryptionkey@0"
I20260925 09:14:36.383586 11832 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 09:14:36.387351 11847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:36.388792 11832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:14:36.388986 11832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "4c7ab7aa78c440208a3fdd68e7798cb1"
format_stamp: "Formatted at 2026-09-25 09:14:36 on dist-test-slave-l6qh"
server_key: "098345ab1bac65dd2b0675fed2e22e99"
server_key_iv: "1fa2170410b427565bc12399aa65b387"
server_key_version: "encryptionkey@0"
I20260925 09:14:36.389213 11832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:36.406970 11832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:36.411604 11832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:36.411931 11832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:36.413034 11832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:36.414500 11832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:36.414579 11832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:36.414667 11832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:36.414727 11832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:36.426206 11689 heartbeater.cc:503] Master 127.3.19.126:43785 was elected leader, sending a full tablet report...
I20260925 09:14:36.449674 11832 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:41505
I20260925 09:14:36.449786 11960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:41505 every 8 connection(s)
I20260925 09:14:36.451107 11832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:14:36.453071 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 11832
I20260925 09:14:36.453279 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:14:36.453722 3149 external_mini_cluster.cc:1428] Setting key 23a96f8131864ff7012c5fd4f8c804b3
Sep 25 09:14:36 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327676, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:14:36.473706 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:36.454080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:35297 (local address 127.3.19.126:43785)
0925 09:14:36.455055 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:36.455063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:36.456057 (+ 994us) server_negotiation.cc:407] Connection header received
0925 09:14:36.458254 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:36.458260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:36.458360 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:36.458471 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:36.460262 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:36.461025 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:36.462962 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:36.463374 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:36.463472 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:36.467331 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:36.467336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:36.467339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:36.467372 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:36.470563 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:36.471297 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:36.471302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:36.471306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:36.471427 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:36.471925 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:36.471930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:36.471933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:36.472205 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:36.472362 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:36.473088 (+ 726us) server_negotiation.cc:299] Negotiation successful
0925 09:14:36.473320 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":638,"threads_started":1}
I20260925 09:14:36.474741 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:36.454768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:35297)
0925 09:14:36.455884 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:36.455945 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:36.457964 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:36.458751 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:36.458760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:36.459384 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:36.460090 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:36.460110 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:36.461182 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:36.461187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:36.462760 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:36.462774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:36.463630 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:36.463636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:36.463877 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:36.464640 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:36.464669 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:36.467088 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:36.470703 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:36.470713 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:36.470739 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:36.471166 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:36.471536 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:14:36.471541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:14:36.471545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:14:36.471803 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:14:36.472329 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:36.472337 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:36.472705 (+ 368us) client_negotiation.cc:769] Sending connection context
0925 09:14:36.472969 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 09:14:36.473550 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":463,"threads_started":1}
I20260925 09:14:36.476181 11961 heartbeater.cc:347] Connected to a master server at 127.3.19.126:43785
I20260925 09:14:36.476480 11961 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:36.477205 11961 heartbeater.cc:511] Master 127.3.19.126:43785 requested a full tablet report, sending...
I20260925 09:14:36.478669 11499 ts_manager.cc:194] Registered new tserver with Master: 4c7ab7aa78c440208a3fdd68e7798cb1 (127.3.19.67:41505)
I20260925 09:14:36.479470 11499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:35297
I20260925 09:14:36.482052 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab.
Entry for principal oryx/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab.
WARNING: no policy specified for oryx/127.3.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab.
Entry for principal oryx/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab.
WARNING: no policy specified for oryx/127.3.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab.
Entry for principal oryx/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab.
I20260925 09:14:36.631399 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 11561
Sep 25 09:14:36 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:14:36.647717 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:36.632469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.3.19.65:44893)
0925 09:14:36.633107 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:36.633115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:36.633145 (+ 30us) server_negotiation.cc:407] Connection header received
0925 09:14:36.633319 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:36.633326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:36.633562 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:36.633720 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:36.635026 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:36.636171 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:36.637504 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:36.638069 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:36.638167 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:36.642090 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:36.642102 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:36.642117 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:36.642156 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:36.645099 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:36.645833 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:36.645839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:36.645842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:36.645966 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:36.646291 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:36.646294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:36.646296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:36.646639 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:36.646868 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:36.647088 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 09:14:36.647286 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":338,"threads_started":1}
I20260925 09:14:36.649000 11684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:36.733453 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 11561
I20260925 09:14:36.749960 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 11695
Sep 25 09:14:36 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:14:36.766593 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:36.750935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.3.19.66:39443)
0925 09:14:36.751517 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:36.751525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:36.751551 (+ 26us) server_negotiation.cc:407] Connection header received
0925 09:14:36.751619 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:36.751625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:36.751842 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:36.751983 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:36.753183 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:36.754272 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:36.755517 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:36.756054 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:36.756150 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:36.760426 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:36.760438 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:36.760453 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:36.760490 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:36.763702 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:36.764529 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:36.764538 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:36.764540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:36.764683 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:36.765022 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:36.765026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:36.765028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:36.765519 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:36.765769 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:36.766055 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 09:14:36.766263 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":324,"threads_started":1}
I20260925 09:14:36.767990 11821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:36.852674 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 11695
I20260925 09:14:36.868893 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 11832
Sep 25 09:14:36 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:14:36.885926 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:36.869910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.3.19.67:41505)
0925 09:14:36.870456 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:36.870464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:36.870524 (+ 60us) server_negotiation.cc:407] Connection header received
0925 09:14:36.870585 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:36.870591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:36.870798 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:36.870924 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:36.872224 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:36.873447 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:36.874797 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:36.875360 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:36.875463 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:36.879942 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:36.879957 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:36.879972 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:36.880013 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:36.882950 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:36.883652 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:36.883662 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:36.883668 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:36.883847 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:36.884239 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:36.884245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:36.884248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:36.884617 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:36.884934 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:36.885306 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 09:14:36.885579 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":249,"threads_started":1}
I20260925 09:14:36.887411 11956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:36.970276 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 11832
I20260925 09:14:36.986559 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 11467
I20260925 09:14:36.987887 11530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:37.088676 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 11467
I20260925 09:14:37.110009 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:43785
--webserver_interface=127.3.19.126
--webserver_port=35451
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:43785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:14:37.444703 11979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:37.445011 11979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:37.445101 11979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:37.452569 11979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:37.452693 11979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:37.452782 11979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:37.452873 11979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:37.452977 11979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:14:37.463238 11979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:43785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:43785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=35451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:37.464697 11979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:37.466091 11979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:37.478570 11987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:37.478618 11985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:37.483572 11984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:37.484325 11979 server_base.cc:1034] running on GCE node
I20260925 09:14:37.485209 11979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:37.486881 11979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:37.488132 11979 hybrid_clock.cc:648] HybridClock initialized: now 1790327677488071 us; error 59 us; skew 500 ppm
I20260925 09:14:37.488511 11979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:37 dist-test-slave-l6qh krb5kdc[11454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327677, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:37.491799 11979 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:14:37.493435 11979 webserver.cc:460] Webserver started at http://127.3.19.126:35451/ using document root <none> and password file <none>
I20260925 09:14:37.494160 11979 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:37.494321 11979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:37.499949 11979 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 09:14:37.502894 11994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:37.504297 11979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 09:14:37.504608 11979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "9d2357cc87144ccab54bab1b5238bc4f"
format_stamp: "Formatted at 2026-09-25 09:14:34 on dist-test-slave-l6qh"
server_key: "6477629d0f8dbb2ecaacf68b7e517510"
server_key_iv: "f3fd41b557a06f590eb8e17b3840925d"
server_key_version: "encryptionkey@0"
I20260925 09:14:37.505674 11979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:37.523478 11979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:37.527971 11979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:37.528261 11979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:37.547912 11979 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:43785
I20260925 09:14:37.547958 12046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:43785 every 8 connection(s)
I20260925 09:14:37.549551 11979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:14:37.553671 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 11979
I20260925 09:14:37.554630 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:44893
--local_ip_for_outbound_sockets=127.3.19.65
--tserver_master_addrs=127.3.19.126:43785
--webserver_port=34749
--webserver_interface=127.3.19.65
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.19.65
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:14:37.557339 12047 sys_catalog.cc:263] Verifying existing consensus state
I20260925 09:14:37.560803 12047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f: Bootstrap starting.
I20260925 09:14:37.589679 12047 log.cc:824] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:37.600937 12047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 09:14:37.601616 12047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f: Bootstrap complete.
I20260925 09:14:37.615149 12047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2357cc87144ccab54bab1b5238bc4f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43785 } }
I20260925 09:14:37.616580 12047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2357cc87144ccab54bab1b5238bc4f, State: Initialized, Role: FOLLOWER
I20260925 09:14:37.617406 12047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2357cc87144ccab54bab1b5238bc4f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43785 } }
I20260925 09:14:37.617697 12047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:14:37.617818 12047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:14:37.617992 12047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 1 FOLLOWER]: Advancing to term 2
I20260925 09:14:37.624316 12047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2357cc87144ccab54bab1b5238bc4f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43785 } }
I20260925 09:14:37.625149 12047 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d2357cc87144ccab54bab1b5238bc4f; no voters:
I20260925 09:14:37.625989 12047 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 09:14:37.626173 12051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 2 FOLLOWER]: Leader election won for term 2
I20260925 09:14:37.627477 12051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [term 2 LEADER]: Becoming Leader. State: Replica: 9d2357cc87144ccab54bab1b5238bc4f, State: Running, Role: LEADER
I20260925 09:14:37.628134 12051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2357cc87144ccab54bab1b5238bc4f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43785 } }
I20260925 09:14:37.628332 12047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:37.630574 12053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2357cc87144ccab54bab1b5238bc4f. Latest consensus state: current_term: 2 leader_uuid: "9d2357cc87144ccab54bab1b5238bc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2357cc87144ccab54bab1b5238bc4f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43785 } } }
I20260925 09:14:37.631093 12053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:37.631472 12052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9d2357cc87144ccab54bab1b5238bc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2357cc87144ccab54bab1b5238bc4f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43785 } } }
I20260925 09:14:37.631845 12052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:37.640053 12057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:14:37.649989 12057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:14:37.651613 12057 catalog_manager.cc:1259] Loaded cluster ID: 723efeae6a4846948553c30b32ca51a5
I20260925 09:14:37.651728 12057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:14:37.655979 12057 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:14:37.657557 12057 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d2357cc87144ccab54bab1b5238bc4f: Loaded TSK: 0
I20260925 09:14:37.659229 12057 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 09:14:37.936381 12049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:37.936735 12049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:37.936908 12049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:37.944589 12049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:37.944751 12049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:37.944922 12049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:14:37.955056 12049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.65.keytab
--principal=oryx/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:44893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=34749
--tserver_master_addrs=127.3.19.126:43785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:37.956514 12049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:37.957885 12049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:37.972324 12075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:37.972389 12077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:37.972651 12074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:37.974102 12049 server_base.cc:1034] running on GCE node
I20260925 09:14:37.974843 12049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:37.975986 12049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:37.977214 12049 hybrid_clock.cc:648] HybridClock initialized: now 1790327677977168 us; error 42 us; skew 500 ppm
I20260925 09:14:37.977609 12049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:37 dist-test-slave-l6qh krb5kdc[11454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327677, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:37.984174 12049 init.cc:377] Logged in from keytab as oryx/127.3.19.65@KRBTEST.COM (short username oryx)
I20260925 09:14:37.986227 12049 webserver.cc:460] Webserver started at http://127.3.19.65:34749/ using document root <none> and password file <none>
I20260925 09:14:37.987207 12049 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:37.987367 12049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:37.995512 12049 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:14:37.999379 12084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:38.001358 12049 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 09:14:38.001588 12049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "d6f1cb51a5b14e39a48f480480284c21"
format_stamp: "Formatted at 2026-09-25 09:14:35 on dist-test-slave-l6qh"
server_key: "0df35554a7c982ed1ea46d64e491f87b"
server_key_iv: "b4725034b3a5ad70c95984d83c404ee6"
server_key_version: "encryptionkey@0"
I20260925 09:14:38.003165 12049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:38.023214 12049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:38.029066 12049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:38.029413 12049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:38.030622 12049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:38.032562 12049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:38.032688 12049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:38.032806 12049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:38.032898 12049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:38.069530 12049 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:44893
I20260925 09:14:38.069646 12197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:44893 every 8 connection(s)
I20260925 09:14:38.071233 12049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:14:38.078204 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 12049
I20260925 09:14:38.079041 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:39443
--local_ip_for_outbound_sockets=127.3.19.66
--tserver_master_addrs=127.3.19.126:43785
--webserver_port=36181
--webserver_interface=127.3.19.66
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.19.66
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:38.091902 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:38.074448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:42839)
0925 09:14:38.075322 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:38.075369 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:38.077089 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:38.078637 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:38.078652 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:38.079505 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:38.080336 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:38.080365 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.082142 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.082150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:38.084321 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:38.084340 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.085546 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.085556 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:38.085908 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:38.086708 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:38.086744 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:38.090781 (+ 4037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":381,"threads_started":1}
W20260925 09:14:38.092608 12198 heartbeater.cc:644] Failed to heartbeat to 127.3.19.126:43785 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.19.126:43785: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
W20260925 09:14:38.092696 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:38.074342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:42839 (local address 127.3.19.126:43785)
0925 09:14:38.075129 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:38.075141 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:38.075506 (+ 365us) server_negotiation.cc:407] Connection header received
0925 09:14:38.077675 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:38.077689 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:38.078038 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:38.078399 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:38.080597 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.081959 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.084494 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.085302 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.085462 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:38.091421 (+ 5959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:42839: BlockingRecv error: recv got EOF from 127.3.19.65:42839 (error 108)
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":347,"threads_started":1}
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:38.102025 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:38.093247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:45645)
0925 09:14:38.093494 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:38.093509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:38.093595 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:38.093932 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:38.093935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:38.094281 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:38.094420 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:38.094429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.096005 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.096009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:38.096884 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:38.096894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.098002 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.098007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:38.098132 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:38.098833 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:38.098849 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:38.101822 (+ 2973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133}
W20260925 09:14:38.102411 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:38.093356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:45645 (local address 127.3.19.126:43785)
0925 09:14:38.093572 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:38.093581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:38.093608 (+ 27us) server_negotiation.cc:407] Connection header received
0925 09:14:38.093683 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:38.093692 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:38.093782 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:38.093907 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:38.094644 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.095847 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.097028 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.097835 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.097970 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:38.102128 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:45645: BlockingRecv error: recv got EOF from 127.3.19.65:45645 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:38.110872 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:38.102819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:34859)
0925 09:14:38.102991 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:38.103006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:38.103116 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:38.103458 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:38.103463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:38.103766 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:38.103905 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:38.103912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.104897 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.104903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:38.105881 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:38.105893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.106618 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.106623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:38.106807 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:38.107413 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:38.107429 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:38.110548 (+ 3119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 09:14:38.111055 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:38.103001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:34859 (local address 127.3.19.126:43785)
0925 09:14:38.103185 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:38.103192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:38.103208 (+ 16us) server_negotiation.cc:407] Connection header received
0925 09:14:38.103255 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:38.103260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:38.103335 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:38.103435 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:38.104048 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.104756 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.106034 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.106472 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.106608 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:38.110787 (+ 4179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:34859: BlockingRecv error: recv got EOF from 127.3.19.65:34859 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 09:14:38.111423 12198 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 09:14:38.416776 12202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:38.417078 12202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:38.417181 12202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:38.424544 12202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:38.424727 12202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:38.424916 12202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:14:38.435051 12202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.66.keytab
--principal=oryx/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:39443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=36181
--tserver_master_addrs=127.3.19.126:43785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:38.436460 12202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:38.437860 12202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:38.451540 12208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:38.451573 12207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:38.451596 12210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:38.452699 12202 server_base.cc:1034] running on GCE node
I20260925 09:14:38.453527 12202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:38.454651 12202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:38.455904 12202 hybrid_clock.cc:648] HybridClock initialized: now 1790327678455859 us; error 59 us; skew 500 ppm
I20260925 09:14:38.456259 12202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327678, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:38.459710 12202 init.cc:377] Logged in from keytab as oryx/127.3.19.66@KRBTEST.COM (short username oryx)
I20260925 09:14:38.461297 12202 webserver.cc:460] Webserver started at http://127.3.19.66:36181/ using document root <none> and password file <none>
I20260925 09:14:38.462020 12202 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:38.462144 12202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:38.468194 12202 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 09:14:38.471131 12217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:38.472616 12202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:14:38.472815 12202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "71da1d8cf52d4bfdb824753a48ee38e6"
format_stamp: "Formatted at 2026-09-25 09:14:35 on dist-test-slave-l6qh"
server_key: "39209fa663f627601b256b6f3f91f044"
server_key_iv: "f410dc07f3acab26efd91eedf83145bf"
server_key_version: "encryptionkey@0"
I20260925 09:14:38.473966 12202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:38.499004 12202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:38.503509 12202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:38.503818 12202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:38.504859 12202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:38.506506 12202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:38.506605 12202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:38.506768 12202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:38.506867 12202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:38.544129 12202 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:39443
I20260925 09:14:38.544184 12330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:39443 every 8 connection(s)
I20260925 09:14:38.545749 12202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:14:38.547351 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 12202
I20260925 09:14:38.548161 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:41505
--local_ip_for_outbound_sockets=127.3.19.67
--tserver_master_addrs=127.3.19.126:43785
--webserver_port=43429
--webserver_interface=127.3.19.67
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.19.67
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 09:14:38.569862 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:38.549022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:56703 (local address 127.3.19.126:43785)
0925 09:14:38.549484 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:38.549494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:38.554222 (+ 4728us) server_negotiation.cc:407] Connection header received
0925 09:14:38.556653 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:38.556659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:38.556772 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:38.556901 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:38.558620 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.559793 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.562353 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.563111 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.563263 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:38.569496 (+ 6233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:56703: BlockingRecv error: recv got EOF from 127.3.19.66:56703 (error 108)
Metrics: {"server-negotiator.queue_time_us":281}
I20260925 09:14:38.570329 12333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:38.549663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:56703)
0925 09:14:38.553999 (+ 4336us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:38.554060 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:38.556321 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:38.557186 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:38.557196 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:38.557831 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:38.558426 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:38.558446 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.560066 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.560072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:38.562163 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:38.562177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.563383 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.563389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:38.563684 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:38.564457 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:38.564494 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:38.569137 (+ 4643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3937,"thread_start_us":3683,"threads_started":1}
W20260925 09:14:38.571122 12331 heartbeater.cc:644] Failed to heartbeat to 127.3.19.126:43785 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.19.126:43785: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:38.580579 12333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:38.571747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:55255)
0925 09:14:38.571942 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:38.571972 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:38.572086 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:38.572439 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:38.572444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:38.572786 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:38.572994 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:38.573003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.574080 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.574086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:38.574990 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:38.575000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.575714 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.575719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:38.575897 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:38.576523 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:38.576541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:38.580248 (+ 3707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 09:14:38.580685 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:38.571867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:55255 (local address 127.3.19.126:43785)
0925 09:14:38.572052 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:38.572058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:38.572077 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:14:38.572224 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:38.572229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:38.572310 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:38.572411 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:38.573135 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.573940 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.575133 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.575559 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.575702 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:38.580445 (+ 4743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:55255: BlockingRecv error: recv got EOF from 127.3.19.66:55255 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:38.590000 12333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:38.581406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:34429)
0925 09:14:38.581588 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:38.581607 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:38.581810 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:38.582145 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:38.582148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:38.582426 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:38.582577 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:38.582584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.584079 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.584083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:38.584995 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:38.585004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.586030 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.586034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:38.586164 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:38.586828 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:38.586855 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:38.589786 (+ 2931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 09:14:38.590384 12331 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 09:14:38.590706 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:38.581588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:34429 (local address 127.3.19.126:43785)
0925 09:14:38.581776 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:38.581783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:38.581804 (+ 21us) server_negotiation.cc:407] Connection header received
0925 09:14:38.581924 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:38.581930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:38.582005 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:38.582110 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:38.582703 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.583931 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.585132 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:38.585866 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:38.586001 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:38.590053 (+ 4052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:34429: BlockingRecv error: recv got EOF from 127.3.19.66:34429 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 09:14:38.895177 12334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:38.895488 12334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:38.895587 12334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:38.902807 12334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:38.902935 12334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:38.903047 12334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:14:38.913369 12334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/krb5kdc/oryx_127.3.19.67.keytab
--principal=oryx/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:41505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=43429
--tserver_master_addrs=127.3.19.126:43785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:38.914778 12334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:38.916013 12334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:38.929359 12339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:38.929464 12340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:38.929376 12342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:38.934741 12334 server_base.cc:1034] running on GCE node
I20260925 09:14:38.935370 12334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:38.936326 12334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:38.937534 12334 hybrid_clock.cc:648] HybridClock initialized: now 1790327678937459 us; error 60 us; skew 500 ppm
I20260925 09:14:38.937865 12334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:14:38 dist-test-slave-l6qh krb5kdc[11454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327678, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:14:38.941022 12334 init.cc:377] Logged in from keytab as oryx/127.3.19.67@KRBTEST.COM (short username oryx)
I20260925 09:14:38.942735 12334 webserver.cc:460] Webserver started at http://127.3.19.67:43429/ using document root <none> and password file <none>
I20260925 09:14:38.943427 12334 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:38.943540 12334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:38.949038 12334 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 09:14:38.951968 12349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:38.953743 12334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:38.953900 12334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "4c7ab7aa78c440208a3fdd68e7798cb1"
format_stamp: "Formatted at 2026-09-25 09:14:36 on dist-test-slave-l6qh"
server_key: "098345ab1bac65dd2b0675fed2e22e99"
server_key_iv: "1fa2170410b427565bc12399aa65b387"
server_key_version: "encryptionkey@0"
I20260925 09:14:38.954898 12334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:38.974648 12334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:38.979763 12334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:38.980016 12334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:38.980988 12334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:38.982353 12334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:38.982438 12334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:38.982527 12334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:38.982579 12334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:39.016932 12334 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:41505
I20260925 09:14:39.016985 12461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:41505 every 8 connection(s)
I20260925 09:14:39.018427 12334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:14:39.026000 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 12334
Sep 25 09:14:39 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:39 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 09:14:39.037678 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:39.021067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:58401 (local address 127.3.19.126:43785)
0925 09:14:39.021331 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:39.021337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:39.022560 (+ 1223us) server_negotiation.cc:407] Connection header received
0925 09:14:39.024358 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:39.024362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:39.024429 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:39.024557 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:39.026316 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.027269 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.029936 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.030640 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.030782 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:39.037339 (+ 6557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:58401: BlockingRecv error: recv got EOF from 127.3.19.67:58401 (error 108)
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":83}
I20260925 09:14:39.038462 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:39.021655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:58401)
0925 09:14:39.022375 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:39.022429 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:39.024074 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:39.024803 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:39.024811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:39.025473 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:39.026128 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:39.026154 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.027468 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.027474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:39.029737 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:39.029752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.030884 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.030890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:39.031104 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:39.031927 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:39.031965 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:39.037001 (+ 5036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":300,"threads_started":1}
W20260925 09:14:39.039228 12462 heartbeater.cc:644] Failed to heartbeat to 127.3.19.126:43785 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.19.126:43785: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
I20260925 09:14:39.044922 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:39.027020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48632 (local address 127.3.19.126:43785)
0925 09:14:39.027718 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:39.027722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:39.027736 (+ 14us) server_negotiation.cc:407] Connection header received
0925 09:14:39.027779 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:39.027782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:39.027831 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:39.027914 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:39.029936 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.031066 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.032939 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.033637 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.033785 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:39.037342 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:39.037355 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:39.037473 (+ 118us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:39.037569 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:39.041413 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:39.042087 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:39.042096 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:39.042105 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:39.042365 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:14:39.042821 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:14:39.042825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:14:39.042827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:14:39.043311 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:39.043572 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:39.044074 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 09:14:39.044478 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":614,"thread_start_us":493,"threads_started":1}
Sep 25 09:14:39 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:39 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:39.049130 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:39.040122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:47547)
0925 09:14:39.040323 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:39.040343 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:39.040446 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:39.040736 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:39.040739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:39.041026 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:39.041167 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:39.041174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.042430 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.042436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:39.043668 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:39.043683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.044686 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.044690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:39.044820 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:39.045462 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:39.045490 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:39.048835 (+ 3345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 09:14:39.049515 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:39.040261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:47547 (local address 127.3.19.126:43785)
0925 09:14:39.040446 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:39.040452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:39.040471 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:14:39.040527 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:39.040532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:39.040603 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:39.040719 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:39.041371 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.042255 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.043846 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.044519 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.044650 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:39.049274 (+ 4624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:47547: BlockingRecv error: recv got EOF from 127.3.19.67:47547 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 09:14:39 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:39 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:39.057516 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:39.049838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:55153)
0925 09:14:39.050006 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:39.050026 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:39.050143 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:39.050466 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:39.050470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:39.050786 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:39.050935 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:39.050943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.051957 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.051963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:39.052874 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:39.052885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.053685 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.053691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:39.053895 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:39.054452 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:39.054468 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:39.057167 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 09:14:39.057655 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:39.049936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:55153 (local address 127.3.19.126:43785)
0925 09:14:39.050137 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:39.050143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:39.050161 (+ 18us) server_negotiation.cc:407] Connection header received
0925 09:14:39.050248 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:39.050253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:39.050337 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:39.050438 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:39.051075 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.051816 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.053020 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.053539 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.053684 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:39.057379 (+ 3695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:55153: BlockingRecv error: recv got EOF from 127.3.19.67:55153 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260925 09:14:39.057938 12462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 09:14:39 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:39 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:39.120394 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:39.111973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:43623)
0925 09:14:39.112478 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:39.112493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:39.112643 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:39.112969 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:39.112973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:39.113492 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:39.113680 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:39.113688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.114685 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.114692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:39.115654 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:39.115664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.116392 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.116399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:39.116577 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:39.117203 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:39.117220 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:39.120056 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":263,"threads_started":1}
W20260925 09:14:39.120502 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:39.112060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:43623 (local address 127.3.19.126:43785)
0925 09:14:39.112283 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:39.112292 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:39.112575 (+ 283us) server_negotiation.cc:407] Connection header received
0925 09:14:39.112782 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:39.112786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:39.112844 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:39.112944 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:39.113823 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.114541 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.115792 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.116212 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.116352 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:39.120274 (+ 3922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:43623: BlockingRecv error: recv got EOF from 127.3.19.65:43623 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 09:14:39 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:39 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:39.599810 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:39.591188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:41347)
0925 09:14:39.591789 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:39.591811 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:39.591970 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:39.592392 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:39.592397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:39.593031 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:39.593223 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:39.593255 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.594266 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.594271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:39.595309 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:39.595321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.596061 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.596066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:39.596205 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:39.596816 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:39.596834 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:39.599584 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":338,"threads_started":1}
W20260925 09:14:39.600152 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:39.591295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:41347 (local address 127.3.19.126:43785)
0925 09:14:39.591558 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:39.591567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:39.591918 (+ 351us) server_negotiation.cc:407] Connection header received
0925 09:14:39.592125 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:39.592131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:39.592222 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:39.592316 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:39.593383 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.594111 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.595460 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:39.595880 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:39.595982 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:39.599806 (+ 3824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:41347: BlockingRecv error: recv got EOF from 127.3.19.66:41347 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 09:14:40 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:40 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:40.067466 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:40.058571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:47343)
0925 09:14:40.059118 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:40.059137 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:40.059267 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:40.059590 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:40.059594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:40.060141 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:40.060377 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:40.060387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:40.061838 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:40.061842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:40.062890 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:40.062902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:40.063623 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:40.063627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:40.063749 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:40.064427 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:40.064444 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:40.067236 (+ 2792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":292,"threads_started":1}
W20260925 09:14:40.067644 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:40.058851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:47343 (local address 127.3.19.126:43785)
0925 09:14:40.059043 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:40.059048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:40.059282 (+ 234us) server_negotiation.cc:407] Connection header received
0925 09:14:40.059378 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:40.059382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:40.059447 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:40.059583 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:40.060544 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:40.061655 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:40.063014 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:40.063441 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:40.063535 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:40.067436 (+ 3901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:47343: BlockingRecv error: recv got EOF from 127.3.19.67:47343 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 09:14:40 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:40 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:40.130555 12470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:40.121579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:42413)
0925 09:14:40.122153 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:40.122175 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:40.122325 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:40.122699 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:40.122703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:40.123257 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:40.123455 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:40.123472 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:40.124893 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:40.124897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:40.125861 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:40.125873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:40.126605 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:40.126609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:40.126723 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:40.127370 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:40.127388 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:40.130312 (+ 2924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":295,"threads_started":1}
W20260925 09:14:40.130854 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:40.121764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:42413 (local address 127.3.19.126:43785)
0925 09:14:40.122000 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:40.122006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:40.122283 (+ 277us) server_negotiation.cc:407] Connection header received
0925 09:14:40.122453 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:40.122460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:40.122545 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:40.122634 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:40.123654 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:40.124747 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:40.126012 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:40.126417 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:40.126560 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:40.130546 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:42413: BlockingRecv error: recv got EOF from 127.3.19.65:42413 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 25 09:14:40 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:40 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:40.609189 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:40.600850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:58915)
0925 09:14:40.601419 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:40.601436 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:40.601595 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:40.601940 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:40.601945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:40.602383 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:40.602524 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:40.602532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:40.603528 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:40.603536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:40.604456 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:40.604467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:40.605173 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:40.605178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:40.605377 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:40.605964 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:40.605983 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:40.608822 (+ 2839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":312,"threads_started":1}
W20260925 09:14:40.609318 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:40.600930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:58915 (local address 127.3.19.126:43785)
0925 09:14:40.601143 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:40.601149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:40.601531 (+ 382us) server_negotiation.cc:407] Connection header received
0925 09:14:40.601725 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:40.601730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:40.601801 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:40.601902 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:40.602666 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:40.603385 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:40.604602 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:40.605014 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:40.605142 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:40.609049 (+ 3907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:58915: BlockingRecv error: recv got EOF from 127.3.19.66:58915 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 25 09:14:41 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:41 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:41.078052 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:41.068584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:38847)
0925 09:14:41.069203 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:41.069251 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:41.069397 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:41.069792 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:41.069797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:41.070291 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:41.070476 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:41.070487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:41.071579 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:41.071585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:41.072694 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:41.072709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:41.073614 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:41.073621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:41.073825 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:41.074624 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:41.074642 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:41.077752 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":395,"threads_started":1}
W20260925 09:14:41.078222 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:41.068835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:38847 (local address 127.3.19.126:43785)
0925 09:14:41.069085 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:41.069093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:41.069337 (+ 244us) server_negotiation.cc:407] Connection header received
0925 09:14:41.069541 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:41.069545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:41.069626 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:41.069728 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:41.070660 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:41.071434 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:41.072853 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:41.073360 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:41.073490 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:41.077971 (+ 4481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:38847: BlockingRecv error: recv got EOF from 127.3.19.67:38847 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 25 09:14:41 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:41 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:41.140170 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:41.131589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:45065)
0925 09:14:41.132138 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:41.132154 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:41.132272 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:41.132575 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:41.132579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:41.132907 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:41.133071 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:41.133079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:41.134383 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:41.134391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:41.135684 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:41.135698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:41.136378 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:41.136383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:41.136505 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:41.137096 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:41.137114 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:41.139907 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":293,"threads_started":1}
W20260925 09:14:41.140375 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:41.131662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:45065 (local address 127.3.19.126:43785)
0925 09:14:41.131876 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:41.131881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:41.132275 (+ 394us) server_negotiation.cc:407] Connection header received
0925 09:14:41.132364 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:41.132368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:41.132431 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:41.132525 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:41.133216 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:41.134204 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:41.135846 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:41.136251 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:41.136350 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:41.140095 (+ 3745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:45065: BlockingRecv error: recv got EOF from 127.3.19.65:45065 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 09:14:41 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:41 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:41.618775 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:41.610239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:39915)
0925 09:14:41.610887 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:41.610905 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:41.611030 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:41.611336 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:41.611340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:41.611726 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:41.611897 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:41.611906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:41.612909 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:41.612912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:41.613990 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:41.614002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:41.614626 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:41.614630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:41.614765 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:41.615405 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:41.615427 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:41.618452 (+ 3025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":360,"threads_started":1}
W20260925 09:14:41.618989 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:41.610328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:39915 (local address 127.3.19.126:43785)
0925 09:14:41.610528 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:41.610533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:41.611013 (+ 480us) server_negotiation.cc:407] Connection header received
0925 09:14:41.611139 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:41.611142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:41.611204 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:41.611288 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:41.612030 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:41.612776 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:41.614110 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:41.614495 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:41.614625 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:41.618741 (+ 4116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:39915: BlockingRecv error: recv got EOF from 127.3.19.66:39915 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 09:14:42 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:42 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:42.088006 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:42.079147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:41665)
0925 09:14:42.079758 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:42.079784 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:42.079920 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:42.080305 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:42.080314 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:42.080689 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:42.080865 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:42.080874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:42.081984 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:42.081987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:42.082862 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:42.082872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:42.083619 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:42.083624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:42.083758 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:42.084479 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:42.084496 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:42.087582 (+ 3086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":318,"threads_started":1}
W20260925 09:14:42.088083 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:42.079235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:41665 (local address 127.3.19.126:43785)
0925 09:14:42.079407 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:42.079414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:42.079884 (+ 470us) server_negotiation.cc:407] Connection header received
0925 09:14:42.080039 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:42.080046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:42.080153 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:42.080233 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:42.081029 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:42.081851 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:42.083002 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:42.083474 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:42.083599 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:42.087818 (+ 4219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:41665: BlockingRecv error: recv got EOF from 127.3.19.67:41665 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 09:14:42 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:42 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:42.150365 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:42.141202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:37951)
0925 09:14:42.141733 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:42.141750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:42.141920 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:42.142336 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:42.142342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:42.142781 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:42.142938 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:42.142948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:42.144035 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:42.144043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:42.145079 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:42.145091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:42.145900 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:42.145907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:42.146129 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:42.146834 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:42.146854 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:42.150022 (+ 3168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":277,"threads_started":1}
W20260925 09:14:42.150489 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:42.141291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:37951 (local address 127.3.19.126:43785)
0925 09:14:42.141515 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:42.141523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:42.141837 (+ 314us) server_negotiation.cc:407] Connection header received
0925 09:14:42.142076 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:42.142082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:42.142159 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:42.142270 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:42.143101 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:42.143886 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:42.145263 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:42.145736 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:42.145885 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:42.150234 (+ 4349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:37951: BlockingRecv error: recv got EOF from 127.3.19.65:37951 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 25 09:14:42 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:42 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:42.629868 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:42.619870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:49771)
0925 09:14:42.620407 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:42.620424 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:42.620630 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:42.621010 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:42.621015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:42.621500 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:42.621666 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:42.621676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:42.622771 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:42.622780 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:42.624187 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:42.624199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:42.625000 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:42.625007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:42.625217 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:42.626079 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:42.626105 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:42.629437 (+ 3332us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":286,"threads_started":1}
W20260925 09:14:42.629976 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:42.619972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:49771 (local address 127.3.19.126:43785)
0925 09:14:42.620218 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:42.620228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:42.620522 (+ 294us) server_negotiation.cc:407] Connection header received
0925 09:14:42.620776 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:42.620782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:42.620860 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:42.620977 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:42.621826 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:42.622632 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:42.624336 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:42.624814 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:42.624973 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:42.629665 (+ 4692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:49771: BlockingRecv error: recv got EOF from 127.3.19.66:49771 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 09:14:43 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:43 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:43.104964 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:43.089107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:45513)
0925 09:14:43.089659 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:43.089677 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:43.089839 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:43.090219 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:43.090224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:43.090632 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:43.090781 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:43.090790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:43.091832 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:43.091837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:43.092812 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:43.092823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:43.093632 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:43.093638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:43.093825 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:43.101838 (+ 8013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:43.101861 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:43.104720 (+ 2859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":310,"threads_started":1}
W20260925 09:14:43.105191 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:43.089260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:45513 (local address 127.3.19.126:43785)
0925 09:14:43.089506 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:43.089513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:43.089768 (+ 255us) server_negotiation.cc:407] Connection header received
0925 09:14:43.089983 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:43.089988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:43.090078 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:43.090190 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:43.090922 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:43.091693 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:43.092953 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:43.093469 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:43.093618 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:43.104957 (+ 11339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:45513: BlockingRecv error: recv got EOF from 127.3.19.67:45513 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 09:14:43 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:43 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 09:14:43.159950 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:43.151536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:41263 (local address 127.3.19.126:43785)
0925 09:14:43.151738 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:43.151743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:43.152037 (+ 294us) server_negotiation.cc:407] Connection header received
0925 09:14:43.152178 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:43.152182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:43.152242 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:43.152330 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:43.153060 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:43.153810 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:43.154952 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:43.155554 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:43.155676 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:43.159787 (+ 4111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:41263: BlockingRecv error: recv got EOF from 127.3.19.65:41263 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 09:14:43.159977 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:43.151386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:41263)
0925 09:14:43.151940 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:43.151957 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:43.152069 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:43.152372 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:43.152376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:43.152763 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:43.152937 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:43.152945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:43.153917 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:43.153921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:43.154827 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:43.154837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:43.155696 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:43.155700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:43.155885 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:43.156619 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:43.156636 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:43.159612 (+ 2976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":290,"threads_started":1}
Sep 25 09:14:43 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:43 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:43.640149 12480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:43.630978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:33921)
0925 09:14:43.631600 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:43.631616 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:43.631779 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:43.632154 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:43.632159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:43.632585 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:43.632762 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:43.632771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:43.633962 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:43.633971 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:43.634987 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:43.635008 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:43.635826 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:43.635832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:43.636045 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:43.636759 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:43.636778 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:43.639749 (+ 2971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":332,"threads_started":1}
W20260925 09:14:43.640340 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:43.631217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:33921 (local address 127.3.19.126:43785)
0925 09:14:43.631455 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:43.631463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:43.631709 (+ 246us) server_negotiation.cc:407] Connection header received
0925 09:14:43.631931 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:43.631936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:43.632007 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:43.632120 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:43.632918 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:43.633815 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:43.635152 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:43.635628 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:43.635812 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:43.639985 (+ 4173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:33921: BlockingRecv error: recv got EOF from 127.3.19.66:33921 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 09:14:44 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:44 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:44.115516 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:44.106147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:45519)
0925 09:14:44.106736 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:44.106757 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:44.106917 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:44.107262 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:44.107267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:44.107730 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:44.107924 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:44.107936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:44.109129 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:44.109135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:44.110760 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:44.110773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:44.111520 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:44.111527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:44.111715 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:44.112297 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:44.112314 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:44.115167 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":276,"threads_started":1}
W20260925 09:14:44.115623 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:44.106212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:45519 (local address 127.3.19.126:43785)
0925 09:14:44.106425 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:44.106432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:44.106845 (+ 413us) server_negotiation.cc:407] Connection header received
0925 09:14:44.107057 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:44.107062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:44.107132 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:44.107236 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:44.108077 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:44.108939 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:44.110911 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:44.111353 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:44.111489 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:44.115377 (+ 3888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:45519: BlockingRecv error: recv got EOF from 127.3.19.67:45519 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 09:14:44 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:44 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:44.169901 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:44.161062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:38693)
0925 09:14:44.161700 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:44.161718 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:44.161848 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:44.162203 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:44.162207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:44.162598 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:44.162759 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:44.162767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:44.163969 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:44.163974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:44.164916 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:44.164927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:44.165754 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:44.165759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:44.165879 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:44.166529 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:44.166547 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:44.169681 (+ 3134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":406,"threads_started":1}
W20260925 09:14:44.170310 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:44.161198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:38693 (local address 127.3.19.126:43785)
0925 09:14:44.161519 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:44.161527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:44.161818 (+ 291us) server_negotiation.cc:407] Connection header received
0925 09:14:44.161981 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:44.161986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:44.162059 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:44.162161 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:44.162924 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:44.163804 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:44.165090 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:44.165603 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:44.165714 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:44.169940 (+ 4226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:38693: BlockingRecv error: recv got EOF from 127.3.19.65:38693 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Sep 25 09:14:44 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:44 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:44.649624 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:44.641295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:42575)
0925 09:14:44.642141 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:44.642157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:44.642268 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:44.642619 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:44.642622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:44.642998 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:44.643160 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:44.643169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:44.644185 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:44.644188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:44.645071 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:44.645080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:44.645822 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:44.645827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:44.645950 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:44.646512 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:44.646528 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:44.649399 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":607,"threads_started":1}
W20260925 09:14:44.649919 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:44.641364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:42575 (local address 127.3.19.126:43785)
0925 09:14:44.641627 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:44.641638 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:44.642258 (+ 620us) server_negotiation.cc:407] Connection header received
0925 09:14:44.642368 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:44.642372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:44.642436 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:44.642522 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:44.643319 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:44.644050 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:44.645262 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:44.645670 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:44.645775 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:44.649633 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:42575: BlockingRecv error: recv got EOF from 127.3.19.66:42575 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 09:14:45 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:45 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:45.125334 12484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:45.116692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:41229)
0925 09:14:45.117279 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:45.117299 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:45.117461 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:45.117910 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:45.117915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:45.118408 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:45.118570 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:45.118579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:45.119559 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:45.119563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:45.120467 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:45.120477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:45.121349 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:45.121354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:45.121482 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:45.122082 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:45.122102 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:45.125032 (+ 2930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":332,"threads_started":1}
W20260925 09:14:45.125546 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:45.116784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:41229 (local address 127.3.19.126:43785)
0925 09:14:45.116976 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:45.116982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:45.117409 (+ 427us) server_negotiation.cc:407] Connection header received
0925 09:14:45.117637 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:45.117643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:45.117743 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:45.117855 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:45.118700 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:45.119431 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:45.120640 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:45.121114 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:45.121211 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:45.125303 (+ 4092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:41229: BlockingRecv error: recv got EOF from 127.3.19.67:41229 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 09:14:45 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:45 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:45.179639 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:45.170847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:34021)
0925 09:14:45.171435 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:45.171450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:45.171609 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:45.171946 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:45.171951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:45.172408 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:45.172567 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:45.172575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:45.173681 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:45.173688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:45.174606 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:45.174617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:45.175293 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:45.175299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:45.175468 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:45.176029 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:45.176046 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:45.179298 (+ 3252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":265,"threads_started":1}
W20260925 09:14:45.179921 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:45.170946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:34021 (local address 127.3.19.126:43785)
0925 09:14:45.171129 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:45.171135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:45.171536 (+ 401us) server_negotiation.cc:407] Connection header received
0925 09:14:45.171736 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:45.171741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:45.171812 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:45.171915 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:45.172725 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:45.173543 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:45.174748 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:45.175148 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:45.175280 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:45.179533 (+ 4253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:34021: BlockingRecv error: recv got EOF from 127.3.19.65:34021 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 09:14:45 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:45 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:45.663249 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:45.650700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:33963)
0925 09:14:45.653095 (+ 2395us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:45.653119 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:45.653294 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:45.653719 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:45.653724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:45.654328 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:45.654581 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:45.654593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:45.656070 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:45.656076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:45.657400 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:45.657417 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:45.658494 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:45.658500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:45.658669 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:45.659415 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:45.659440 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:45.662871 (+ 3431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2250,"thread_start_us":2028,"threads_started":1}
W20260925 09:14:45.663331 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:45.650737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:33963 (local address 127.3.19.126:43785)
0925 09:14:45.650975 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:45.650985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:45.653288 (+ 2303us) server_negotiation.cc:407] Connection header received
0925 09:14:45.653422 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:45.653461 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:45.653566 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:45.653691 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:45.654744 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:45.655885 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:45.657660 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:45.658316 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:45.658467 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:45.663063 (+ 4596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:33963: BlockingRecv error: recv got EOF from 127.3.19.66:33963 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 09:14:46 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:46 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:46.135547 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:46.126522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:44159)
0925 09:14:46.127469 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:46.127491 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:46.127615 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:46.127956 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:46.127961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:46.128490 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:46.128722 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:46.128736 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:46.129873 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:46.129882 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:46.130808 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:46.130819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:46.131580 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:46.131586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:46.131756 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:46.132387 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:46.132404 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:46.135188 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":629,"threads_started":1}
W20260925 09:14:46.135640 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:46.126707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:44159 (local address 127.3.19.126:43785)
0925 09:14:46.126903 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:46.126911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:46.127624 (+ 713us) server_negotiation.cc:407] Connection header received
0925 09:14:46.127739 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:46.127742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:46.127810 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:46.127923 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:46.128876 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:46.129731 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:46.130953 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:46.131411 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:46.131548 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:46.135407 (+ 3859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:44159: BlockingRecv error: recv got EOF from 127.3.19.67:44159 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 09:14:46 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:46 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:46.189656 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:46.180535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:52415)
0925 09:14:46.181160 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:46.181180 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:46.181374 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:46.181776 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:46.181782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:46.182252 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:46.182414 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:46.182424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:46.183547 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:46.183553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:46.184529 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:46.184541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:46.185420 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:46.185427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:46.185636 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:46.186294 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:46.186313 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:46.189272 (+ 2959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":306,"threads_started":1}
W20260925 09:14:46.189893 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:46.180875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:52415 (local address 127.3.19.126:43785)
0925 09:14:46.181106 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:46.181115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:46.181287 (+ 172us) server_negotiation.cc:407] Connection header received
0925 09:14:46.181548 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:46.181554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:46.181629 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:46.181741 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:46.182582 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:46.183369 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:46.184691 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:46.185172 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:46.185373 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:46.189494 (+ 4121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:52415: BlockingRecv error: recv got EOF from 127.3.19.65:52415 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 25 09:14:46 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:46 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:46.673365 12489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:46.664356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:56013)
0925 09:14:46.664896 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:46.664912 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:46.665023 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:46.665385 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:46.665389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:46.665819 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:46.665979 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:46.665989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:46.667216 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:46.667220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:46.668167 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:46.668179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:46.668890 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:46.668895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:46.669015 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:46.669750 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:46.669768 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:46.673048 (+ 3280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":289,"threads_started":1}
W20260925 09:14:46.673666 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:46.664532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:56013 (local address 127.3.19.126:43785)
0925 09:14:46.664697 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:46.664701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:46.664973 (+ 272us) server_negotiation.cc:407] Connection header received
0925 09:14:46.665126 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:46.665130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:46.665190 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:46.665290 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:46.666129 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:46.667063 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:46.668304 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:46.668721 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:46.668815 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:46.673361 (+ 4546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:56013: BlockingRecv error: recv got EOF from 127.3.19.66:56013 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 09:14:47 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:47 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:47.146797 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:47.136726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:54021)
0925 09:14:47.137374 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:47.137395 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:47.137517 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:47.138067 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:47.138071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:47.138489 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:47.138697 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:47.138706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:47.139751 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:47.139755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:47.141414 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:47.141429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:47.142577 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:47.142582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:47.142725 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:47.143322 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:47.143340 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:47.146462 (+ 3122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":374,"threads_started":1}
W20260925 09:14:47.147011 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:47.136839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:54021 (local address 127.3.19.126:43785)
0925 09:14:47.137098 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:47.137104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:47.137483 (+ 379us) server_negotiation.cc:407] Connection header received
0925 09:14:47.137798 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:47.137802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:47.137867 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:47.137968 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:47.138848 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:47.139599 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:47.141639 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:47.142375 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:47.142493 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:47.146714 (+ 4221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:54021: BlockingRecv error: recv got EOF from 127.3.19.67:54021 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 25 09:14:47 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:47 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:47.200348 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:47.190845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:33523)
0925 09:14:47.191427 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:47.191449 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:47.191566 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:47.192020 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:47.192026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:47.192669 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:47.192933 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:47.192948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:47.194264 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:47.194268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:47.195316 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:47.195327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:47.196054 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:47.196059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:47.196201 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:47.196915 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:47.196934 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:47.199989 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":325,"threads_started":1}
W20260925 09:14:47.200460 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:47.190951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:33523 (local address 127.3.19.126:43785)
0925 09:14:47.191245 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:47.191254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:47.191547 (+ 293us) server_negotiation.cc:407] Connection header received
0925 09:14:47.191733 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:47.191740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:47.191827 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:47.191935 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:47.193115 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:47.194132 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:47.195457 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:47.195888 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:47.195991 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:47.200236 (+ 4245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:33523: BlockingRecv error: recv got EOF from 127.3.19.65:33523 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 25 09:14:47 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:47 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:47.683643 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:47.674297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:48677)
0925 09:14:47.674949 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:47.674994 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:47.675139 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:47.675516 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:47.675520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:47.676090 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:47.676269 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:47.676278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:47.677395 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:47.677398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:47.678350 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:47.678361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:47.679548 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:47.679554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:47.679776 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:47.680515 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:47.680532 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:47.683401 (+ 2869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":359,"threads_started":1}
W20260925 09:14:47.683948 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:47.674392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:48677 (local address 127.3.19.126:43785)
0925 09:14:47.674621 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:47.674630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:47.675077 (+ 447us) server_negotiation.cc:407] Connection header received
0925 09:14:47.675273 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:47.675283 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:47.675361 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:47.675468 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:47.676388 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:47.677175 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:47.678468 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:47.679354 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:47.679514 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:47.683643 (+ 4129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:48677: BlockingRecv error: recv got EOF from 127.3.19.66:48677 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 09:14:48 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:48 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:48.161751 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:48.147888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:57695)
0925 09:14:48.148428 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:48.148449 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:48.148574 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:48.149076 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:48.149081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:48.149587 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:48.149858 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:48.149871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:48.151171 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:48.151176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:48.152144 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:48.152156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:48.152821 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:48.152826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:48.152950 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:48.158109 (+ 5159us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:48.158130 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:48.161392 (+ 3262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":310,"threads_started":1}
W20260925 09:14:48.161895 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:48.147941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:57695 (local address 127.3.19.126:43785)
0925 09:14:48.148110 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:48.148114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:48.148528 (+ 414us) server_negotiation.cc:407] Connection header received
0925 09:14:48.148724 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:48.148729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:48.148807 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:48.148927 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:48.150023 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:48.151032 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:48.152268 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:48.152676 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:48.152776 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:48.161637 (+ 8861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:57695: BlockingRecv error: recv got EOF from 127.3.19.67:57695 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 09:14:48 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:48 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:48.210445 12494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:48.201533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:35461)
0925 09:14:48.202125 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:48.202141 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:48.202300 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:48.202687 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:48.202692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:48.203107 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:48.203248 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:48.203257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:48.204310 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:48.204316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:48.205336 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:48.205348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:48.206101 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:48.206106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:48.206291 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:48.206880 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:48.206898 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:48.210192 (+ 3294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":337,"threads_started":1}
W20260925 09:14:48.210719 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:48.201726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:35461 (local address 127.3.19.126:43785)
0925 09:14:48.201971 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:48.201978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:48.202230 (+ 252us) server_negotiation.cc:407] Connection header received
0925 09:14:48.202439 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:48.202445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:48.202521 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:48.202656 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:48.203399 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:48.204167 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:48.205469 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:48.205926 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:48.206069 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:48.210445 (+ 4376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:35461: BlockingRecv error: recv got EOF from 127.3.19.65:35461 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 09:14:48 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:48 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:48.693003 12495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:48.684596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:53643)
0925 09:14:48.685100 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:48.685117 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:48.685298 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:48.685650 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:48.685654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:48.686065 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:48.686212 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:48.686220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:48.687274 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:48.687279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:48.688362 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:48.688374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:48.689078 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:48.689083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:48.689324 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:48.689903 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:48.689919 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:48.692720 (+ 2801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":285,"threads_started":1}
W20260925 09:14:48.693301 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:48.684663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:53643 (local address 127.3.19.126:43785)
0925 09:14:48.684897 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:48.684907 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:48.685207 (+ 300us) server_negotiation.cc:407] Connection header received
0925 09:14:48.685426 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:48.685432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:48.685510 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:48.685625 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:48.686360 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:48.687117 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:48.688508 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:48.688925 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:48.689062 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:48.692979 (+ 3917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:53643: BlockingRecv error: recv got EOF from 127.3.19.66:53643 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 09:14:49 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:49 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:49.171414 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:49.162922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:58591)
0925 09:14:49.163490 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:49.163512 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:49.163672 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:49.164090 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:49.164096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:49.164596 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:49.164800 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:49.164812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:49.165976 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:49.165980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:49.166994 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:49.167006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:49.167711 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:49.167716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:49.167847 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:49.168515 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:49.168532 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:49.171189 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":344,"threads_started":1}
W20260925 09:14:49.171693 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:49.163138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:58591 (local address 127.3.19.126:43785)
0925 09:14:49.163389 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:49.163394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:49.163604 (+ 210us) server_negotiation.cc:407] Connection header received
0925 09:14:49.163797 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:49.163802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:49.163901 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:49.164002 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:49.164965 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:49.165827 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:49.167133 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:49.167545 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:49.167647 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:49.171395 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:58591: BlockingRecv error: recv got EOF from 127.3.19.67:58591 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Sep 25 09:14:49 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:49 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:49.220002 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:49.211525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:43843)
0925 09:14:49.212005 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:49.212021 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:49.212151 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:49.212504 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:49.212512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:49.212929 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:49.213114 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:49.213123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:49.214161 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:49.214167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:49.215072 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:49.215083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:49.215966 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:49.215972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:49.216155 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:49.216769 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:49.216787 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:49.219754 (+ 2967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":272,"threads_started":1}
W20260925 09:14:49.220196 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:49.211682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:43843 (local address 127.3.19.126:43785)
0925 09:14:49.211928 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:49.211935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:49.212108 (+ 173us) server_negotiation.cc:407] Connection header received
0925 09:14:49.212266 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:49.212271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:49.212351 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:49.212462 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:49.213282 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:49.214028 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:49.215217 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:49.215738 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:49.215950 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:49.219964 (+ 4014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:43843: BlockingRecv error: recv got EOF from 127.3.19.65:43843 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 09:14:49 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:49 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:49.702672 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:49.694190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:50491)
0925 09:14:49.694733 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:49.694749 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:49.694903 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:49.695238 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:49.695246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:49.695665 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:49.695852 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:49.695860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:49.696910 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:49.696916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:49.697885 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:49.697896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:49.698588 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:49.698593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:49.698759 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:49.699360 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:49.699377 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:49.702317 (+ 2940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":310,"threads_started":1}
W20260925 09:14:49.702785 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:49.694158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:50491 (local address 127.3.19.126:43785)
0925 09:14:49.694330 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:49.694336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:49.694835 (+ 499us) server_negotiation.cc:407] Connection header received
0925 09:14:49.695029 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:49.695039 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:49.695102 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:49.695198 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:49.695993 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:49.696767 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:49.698024 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:49.698437 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:49.698575 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:49.702527 (+ 3952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:50491: BlockingRecv error: recv got EOF from 127.3.19.66:50491 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 09:14:50 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:50 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:50.181934 12499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:50.172440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:53039)
0925 09:14:50.173046 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:50.173075 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:50.173222 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:50.173699 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:50.173706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:50.174263 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:50.174442 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:50.174451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:50.175901 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:50.175906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:50.176819 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:50.176830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:50.177585 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:50.177590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:50.177717 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:50.178459 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:50.178476 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:50.181692 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":356,"threads_started":1}
W20260925 09:14:50.182289 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:50.172529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:53039 (local address 127.3.19.126:43785)
0925 09:14:50.172759 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:50.172768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:50.173163 (+ 395us) server_negotiation.cc:407] Connection header received
0925 09:14:50.173411 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:50.173417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:50.173538 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:50.173621 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:50.174785 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:50.175762 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:50.176969 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:50.177417 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:50.177542 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:50.181957 (+ 4415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:53039: BlockingRecv error: recv got EOF from 127.3.19.67:53039 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 09:14:50 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:50 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:50.244614 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:50.221371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:48253)
0925 09:14:50.233329 (+ 11958us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:50.233362 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:50.233572 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:50.233999 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:50.234006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:50.234636 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:50.234903 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:50.234922 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:50.236307 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:50.236316 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:50.238033 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:50.238078 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:50.238966 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:50.238985 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:50.239221 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:50.240149 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:50.240199 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:50.244215 (+ 4016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":11736,"thread_start_us":12305,"threads_started":1}
W20260925 09:14:50.245076 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:50.221920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:48253 (local address 127.3.19.126:43785)
0925 09:14:50.222530 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:50.222542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:50.233503 (+ 10961us) server_negotiation.cc:407] Connection header received
0925 09:14:50.233741 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:50.233748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:50.233843 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:50.233951 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:50.235114 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:50.236135 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:50.238254 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:50.238802 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:50.238940 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:50.244488 (+ 5548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:48253: BlockingRecv error: recv got EOF from 127.3.19.65:48253 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 09:14:50 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:50 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:50.712783 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:50.703802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:53861)
0925 09:14:50.704298 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:50.704338 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:50.704497 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:50.704873 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:50.704877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:50.705330 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:50.705482 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:50.705490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:50.706477 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:50.706485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:50.707406 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:50.707417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:50.708242 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:50.708247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:50.708453 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:50.709193 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:50.709211 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:50.712434 (+ 3223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":276,"threads_started":1}
W20260925 09:14:50.712867 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:50.703883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:53861 (local address 127.3.19.126:43785)
0925 09:14:50.704133 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:50.704141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:50.704500 (+ 359us) server_negotiation.cc:407] Connection header received
0925 09:14:50.704674 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:50.704678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:50.704740 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:50.704840 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:50.705634 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:50.706340 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:50.707552 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:50.708105 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:50.708232 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:50.712647 (+ 4415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:53861: BlockingRecv error: recv got EOF from 127.3.19.66:53861 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 09:14:51 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:51 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:51.191282 12502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:51.182887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:49209)
0925 09:14:51.183458 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:51.183480 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:51.183605 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:51.183922 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:51.183925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:51.184319 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:51.184467 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:51.184475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:51.185566 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:51.185570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:51.186492 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:51.186502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:51.187284 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:51.187288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:51.187417 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:51.188090 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:51.188111 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:51.191047 (+ 2936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":352,"threads_started":1}
W20260925 09:14:51.191468 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:51.182995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:49209 (local address 127.3.19.126:43785)
0925 09:14:51.183168 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:51.183177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:51.183570 (+ 393us) server_negotiation.cc:407] Connection header received
0925 09:14:51.183709 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:51.183713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:51.183778 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:51.183865 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:51.184629 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:51.185413 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:51.186659 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:51.187129 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:51.187245 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:51.191247 (+ 4002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:49209: BlockingRecv error: recv got EOF from 127.3.19.67:49209 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 09:14:51 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:51 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:51.254364 12503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:51.245758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:38515)
0925 09:14:51.246382 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:51.246399 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:51.246532 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:51.246878 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:51.246882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:51.247273 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:51.247429 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:51.247438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:51.248542 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:51.248546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:51.249610 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:51.249624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:51.250396 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:51.250402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:51.250555 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:51.251230 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:51.251248 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:51.254130 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":342,"threads_started":1}
W20260925 09:14:51.254545 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:51.245809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:38515 (local address 127.3.19.126:43785)
0925 09:14:51.245999 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:51.246003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:51.246487 (+ 484us) server_negotiation.cc:407] Connection header received
0925 09:14:51.246635 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:51.246639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:51.246701 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:51.246788 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:51.247597 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:51.248387 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:51.249753 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:51.250199 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:51.250299 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:51.254311 (+ 4012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:38515: BlockingRecv error: recv got EOF from 127.3.19.65:38515 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 09:14:51 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:51 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:51.722581 12504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:51.713901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:36667)
0925 09:14:51.714473 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:51.714488 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:51.714634 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:51.714982 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:51.714986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:51.715410 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:51.715568 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:51.715576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:51.716664 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:51.716670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:51.717635 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:51.717647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:51.718420 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:51.718426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:51.718604 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:51.719275 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:51.719294 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:51.722222 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":289,"threads_started":1}
W20260925 09:14:51.722756 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:51.714085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:36667 (local address 127.3.19.126:43785)
0925 09:14:51.714318 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:51.714333 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:51.714573 (+ 240us) server_negotiation.cc:407] Connection header received
0925 09:14:51.714770 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:51.714775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:51.714854 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:51.714952 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:51.715719 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:51.716517 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:51.717789 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:51.718237 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:51.718382 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:51.722498 (+ 4116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:36667: BlockingRecv error: recv got EOF from 127.3.19.66:36667 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 25 09:14:52 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:52 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:52.200397 12505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:52.192190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:54543)
0925 09:14:52.192714 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:52.192733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:52.192845 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:52.193182 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:52.193186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:52.193611 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:52.193788 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:52.193797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:52.194830 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:52.194834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:52.195764 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:52.195774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:52.196526 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:52.196530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:52.196670 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:52.197279 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:52.197296 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:52.200161 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":279,"threads_started":1}
W20260925 09:14:52.200707 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:52.192310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:54543 (local address 127.3.19.126:43785)
0925 09:14:52.192468 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:52.192472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:52.192849 (+ 377us) server_negotiation.cc:407] Connection header received
0925 09:14:52.192955 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:52.192959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:52.193043 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:52.193140 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:52.193940 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:52.194706 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:52.195894 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:52.196355 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:52.196466 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:52.200377 (+ 3911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:54543: BlockingRecv error: recv got EOF from 127.3.19.67:54543 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 09:14:52 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:52 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:52.263860 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:52.255285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:48509)
0925 09:14:52.255806 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:52.255823 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:52.255975 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:52.256319 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:52.256325 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:52.256697 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:52.256878 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:52.256886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:52.257962 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:52.257968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:52.258931 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:52.258942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:52.259697 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:52.259703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:52.259878 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:52.260498 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:52.260516 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:52.263522 (+ 3006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":272,"threads_started":1}
W20260925 09:14:52.264091 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:52.255367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:48509 (local address 127.3.19.126:43785)
0925 09:14:52.255678 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:52.255684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:52.255902 (+ 218us) server_negotiation.cc:407] Connection header received
0925 09:14:52.256110 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:52.256115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:52.256173 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:52.256272 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:52.257013 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:52.257826 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:52.259064 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:52.259555 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:52.259687 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:52.263808 (+ 4121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:48509: BlockingRecv error: recv got EOF from 127.3.19.65:48509 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
Sep 25 09:14:52 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:52 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:52.732138 12507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:52.723626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:35477)
0925 09:14:52.724072 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:52.724097 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:52.724223 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:52.724526 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:52.724529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:52.724890 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:52.725061 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:52.725069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:52.726127 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:52.726131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:52.727029 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:52.727038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:52.727733 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:52.727739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:52.727889 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:52.728654 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:52.728675 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:52.731764 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":225,"threads_started":1}
W20260925 09:14:52.732198 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:52.723717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:35477 (local address 127.3.19.126:43785)
0925 09:14:52.723955 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:52.723962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:52.724188 (+ 226us) server_negotiation.cc:407] Connection header received
0925 09:14:52.724324 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:52.724327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:52.724387 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:52.724467 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:52.725211 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:52.725977 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:52.727149 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:52.727569 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:52.727660 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:52.731986 (+ 4326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:35477: BlockingRecv error: recv got EOF from 127.3.19.66:35477 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 25 09:14:53 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:53 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.67@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:53.219837 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:53.201396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.67:40755)
0925 09:14:53.201901 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:53.201923 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:53.202093 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:53.202427 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:53.202432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:53.202814 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:53.202990 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:53.202998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:53.204014 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:53.204020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:53.205039 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:53.205052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:53.205982 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:53.205989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:53.206127 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:53.216159 (+ 10032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:53.216182 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:53.219468 (+ 3286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":291,"threads_started":1}
W20260925 09:14:53.219955 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:53.201531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:40755 (local address 127.3.19.126:43785)
0925 09:14:53.201810 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:53.201816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:53.202005 (+ 189us) server_negotiation.cc:407] Connection header received
0925 09:14:53.202227 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:53.202237 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:53.202296 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:53.202396 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:53.203146 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:53.203870 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:53.205252 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:53.205807 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:53.205974 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:53.219714 (+ 13740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.67:40755: BlockingRecv error: recv got EOF from 127.3.19.67:40755 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 09:14:53 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:53 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.65@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:53.274168 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:53.265203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.65:36133)
0925 09:14:53.265768 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:53.265786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:53.265900 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:53.266510 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:53.266515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:53.267012 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:53.267268 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:53.267278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:53.268630 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:53.268634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:53.269613 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:53.269626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:53.270312 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:53.270316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:53.270435 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:53.271038 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:53.271061 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:53.273903 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":299,"threads_started":1}
W20260925 09:14:53.274385 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:53.265353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:36133 (local address 127.3.19.126:43785)
0925 09:14:53.265562 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:53.265567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:53.265874 (+ 307us) server_negotiation.cc:407] Connection header received
0925 09:14:53.266071 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:53.266075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:53.266172 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:53.266297 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:53.267418 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:53.268491 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:53.269745 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:53.270183 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:53.270277 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:53.274125 (+ 3848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.65:36133: BlockingRecv error: recv got EOF from 127.3.19.65:36133 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 09:14:53 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 09:14:53 dist-test-slave-l6qh krb5kdc[11454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.19.66@KRBTEST.COM for oryx/127.3.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 09:14:53.741968 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:53.733429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43785 (local address 127.3.19.66:42561)
0925 09:14:53.733950 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:53.733966 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:53.734081 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:53.734461 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:53.734464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:53.734992 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:53.735184 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:53.735193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:53.736215 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:53.736224 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:53.737161 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:53.737172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:53.737945 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:53.737951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:53.738118 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:53.738698 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:14:53.738714 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:53.741591 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.19.126:43785: Server oryx/127.3.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":299,"threads_started":1}
W20260925 09:14:53.742100 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:53.733569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:42561 (local address 127.3.19.126:43785)
0925 09:14:53.733784 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:53.733789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:53.734054 (+ 265us) server_negotiation.cc:407] Connection header received
0925 09:14:53.734214 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:53.734220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:53.734307 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:53.734443 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:53.735334 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:53.736067 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:53.737342 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:53.737800 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:53.737936 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:53.741790 (+ 3854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.19.66:42561: BlockingRecv error: recv got EOF from 127.3.19.66:42561 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 09:14:54.031039 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 12049
I20260925 09:14:54.044451 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:54.032325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.3.19.65:44893)
0925 09:14:54.032897 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:54.032905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:54.032934 (+ 29us) server_negotiation.cc:407] Connection header received
0925 09:14:54.032995 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:54.033002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:54.033283 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:54.033463 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:54.035117 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:54.036327 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:54.037845 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:54.038470 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:54.038584 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:54.041294 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:54.041310 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:54.041326 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:54.041371 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:54.043918 (+ 2547us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.19.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 09:14:54.044220 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48048: Request ticket server kudu/127.3.19.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":274,"threads_started":1}
W20260925 09:14:54.044584 12513 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48048: Request ticket server kudu/127.3.19.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 09:14:54.044755 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 12049: Not authorized: Client connection negotiation failed: client connection to 127.3.19.65:44893: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.19.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 09:14:54.044945 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 12049
I20260925 09:14:54.062595 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 12202
I20260925 09:14:54.075872 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:54.063876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.3.19.66:39443)
0925 09:14:54.064499 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:54.064507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:54.064544 (+ 37us) server_negotiation.cc:407] Connection header received
0925 09:14:54.064606 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:54.064613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:54.064864 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:54.065046 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:54.066446 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:54.067586 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:54.068912 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:54.069674 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:54.069788 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:54.072375 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:54.072385 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:54.072399 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:54.072437 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:54.075252 (+ 2815us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.19.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 09:14:54.075553 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49236: Request ticket server kudu/127.3.19.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":316,"threads_started":1}
W20260925 09:14:54.076006 12514 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49236: Request ticket server kudu/127.3.19.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 09:14:54.076131 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 12202: Not authorized: Client connection negotiation failed: client connection to 127.3.19.66:39443: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.19.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 09:14:54.076284 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 12202
I20260925 09:14:54.093065 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 12334
I20260925 09:14:54.106364 12515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:54.094119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.3.19.67:41505)
0925 09:14:54.094732 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:54.094739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:54.094767 (+ 28us) server_negotiation.cc:407] Connection header received
0925 09:14:54.094828 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:54.094835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:54.095054 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:54.095222 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:54.096637 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:54.097827 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:54.099250 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:54.099834 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:54.099938 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:54.102834 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:54.102844 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:54.102860 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:14:54.102898 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:54.105800 (+ 2902us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.19.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 09:14:54.106105 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36740: Request ticket server kudu/127.3.19.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":318,"threads_started":1}
W20260925 09:14:54.106503 12515 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36740: Request ticket server kudu/127.3.19.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 09:14:54.106662 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 12334: Not authorized: Client connection negotiation failed: client connection to 127.3.19.67:41505: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.19.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 09:14:54.106855 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 12334
I20260925 09:14:54.124287 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 11979
I20260925 09:14:54.125633 12042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:54.239173 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 11979
2026-09-25T09:14:54Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21323 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T09:14:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:14:54Z Disabled control of system clock
I20260925 09:14:54.323827 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:36383
--webserver_interface=127.3.19.126
--webserver_port=0
--builtin_ntp_servers=127.3.19.84:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:36383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 09:14:54.664084 12520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:54.664412 12520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:54.664494 12520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:54.671911 12520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:54.672034 12520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:54.672073 12520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:54.672124 12520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:54.672168 12520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:14:54.681636 12520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:36383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:36383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:54.682938 12520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:54.684643 12520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:54.696794 12525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:54.696831 12528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:54.696794 12526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:54.697928 12520 server_base.cc:1034] running on GCE node
I20260925 09:14:54.698783 12520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:54.700258 12520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:54.701511 12520 hybrid_clock.cc:648] HybridClock initialized: now 1790327694701456 us; error 62 us; skew 500 ppm
I20260925 09:14:54.701908 12520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:54.704264 12520 webserver.cc:460] Webserver started at http://127.3.19.126:41067/ using document root <none> and password file <none>
I20260925 09:14:54.705015 12520 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:54.705145 12520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:54.705518 12520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:54.708582 12520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "741928f2f09b45cb89b0d50091578b6c"
format_stamp: "Formatted at 2026-09-25 09:14:54 on dist-test-slave-l6qh"
server_key: "bd9bfd3f83caa5e5bc41c8413a531bc0"
server_key_iv: "c88199255f4876784fe2121efe5e0c6a"
server_key_version: "encryptionkey@0"
I20260925 09:14:54.709439 12520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "741928f2f09b45cb89b0d50091578b6c"
format_stamp: "Formatted at 2026-09-25 09:14:54 on dist-test-slave-l6qh"
server_key: "bd9bfd3f83caa5e5bc41c8413a531bc0"
server_key_iv: "c88199255f4876784fe2121efe5e0c6a"
server_key_version: "encryptionkey@0"
I20260925 09:14:54.714993 12520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 09:14:54.719177 12534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:54.720695 12520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:14:54.720904 12520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "741928f2f09b45cb89b0d50091578b6c"
format_stamp: "Formatted at 2026-09-25 09:14:54 on dist-test-slave-l6qh"
server_key: "bd9bfd3f83caa5e5bc41c8413a531bc0"
server_key_iv: "c88199255f4876784fe2121efe5e0c6a"
server_key_version: "encryptionkey@0"
I20260925 09:14:54.721143 12520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:54.744825 12520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:54.745796 12520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:54.746116 12520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:54.765992 12520 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:36383
I20260925 09:14:54.766031 12585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:36383 every 8 connection(s)
I20260925 09:14:54.767375 12520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:14:54.768468 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 12520
I20260925 09:14:54.768682 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:14:54.769452 3149 external_mini_cluster.cc:1428] Setting key 97b1d715a9e08fcf966be26b107931ea
I20260925 09:14:54.772128 12586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:54.783244 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:54.771472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.3.19.126:36383)
0925 09:14:54.772526 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:54.772537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:54.772580 (+ 43us) server_negotiation.cc:407] Connection header received
0925 09:14:54.774258 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:54.774280 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:54.774742 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:54.775151 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:54.775828 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:54.777191 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:54.778559 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:54.779255 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:54.779446 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:54.780734 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:54.780741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:54.780759 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:54.781408 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:54.781486 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:54.781506 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:54.781674 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:54.781916 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 09:14:54.782229 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":347,"threads_started":1}
I20260925 09:14:54.784834 12586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c: Bootstrap starting.
I20260925 09:14:54.788134 12586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:54.789121 12586 log.cc:824] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:54.792258 12586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c: No bootstrap required, opened a new log
I20260925 09:14:54.799755 12586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741928f2f09b45cb89b0d50091578b6c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36383 } }
I20260925 09:14:54.800278 12586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:54.800431 12586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741928f2f09b45cb89b0d50091578b6c, State: Initialized, Role: FOLLOWER
I20260925 09:14:54.801026 12586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741928f2f09b45cb89b0d50091578b6c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36383 } }
I20260925 09:14:54.801270 12586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:14:54.801414 12586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:14:54.801575 12586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:54.803753 12586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741928f2f09b45cb89b0d50091578b6c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36383 } }
I20260925 09:14:54.804280 12586 leader_election.cc:302] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 741928f2f09b45cb89b0d50091578b6c; no voters:
I20260925 09:14:54.804901 12586 leader_election.cc:288] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:14:54.805076 12591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:54.805838 12591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [term 1 LEADER]: Becoming Leader. State: Replica: 741928f2f09b45cb89b0d50091578b6c, State: Running, Role: LEADER
I20260925 09:14:54.806416 12591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741928f2f09b45cb89b0d50091578b6c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36383 } }
I20260925 09:14:54.807122 12586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:54.809512 12592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "741928f2f09b45cb89b0d50091578b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741928f2f09b45cb89b0d50091578b6c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36383 } } }
I20260925 09:14:54.810015 12592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:54.809598 12593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 741928f2f09b45cb89b0d50091578b6c. Latest consensus state: current_term: 1 leader_uuid: "741928f2f09b45cb89b0d50091578b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741928f2f09b45cb89b0d50091578b6c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36383 } } }
I20260925 09:14:54.810487 12593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:54.812301 12599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:14:54.818005 12599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:14:54.827687 12599 catalog_manager.cc:1347] Generated new cluster ID: c5954dc2708046d8970c570667d0c02f
I20260925 09:14:54.827836 12599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:14:54.848404 12599 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:14:54.849918 12599 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:14:54.860626 12599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 741928f2f09b45cb89b0d50091578b6c: Generated new TSK 0
I20260925 09:14:54.861586 12599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:14:54.870198 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:36383
--builtin_ntp_servers=127.3.19.84:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 09:14:55.216451 12610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:55.216734 12610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:55.216863 12610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:55.224555 12610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:55.224664 12610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:55.224814 12610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:14:55.234324 12610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:36383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:55.235647 12610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:55.237500 12610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:55.250916 12616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:55.250880 12618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:55.250973 12615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:55.251461 12610 server_base.cc:1034] running on GCE node
I20260925 09:14:55.252332 12610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:55.253434 12610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:55.254693 12610 hybrid_clock.cc:648] HybridClock initialized: now 1790327695254625 us; error 53 us; skew 500 ppm
I20260925 09:14:55.255043 12610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:55.257592 12610 webserver.cc:460] Webserver started at http://127.3.19.65:34725/ using document root <none> and password file <none>
I20260925 09:14:55.258322 12610 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:55.258450 12610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:55.258780 12610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:55.261737 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "b82953dea67d413c94ef533426d8ad08"
format_stamp: "Formatted at 2026-09-25 09:14:55 on dist-test-slave-l6qh"
server_key: "3b2b77688be618bc281506b30c05b58b"
server_key_iv: "c43908f051192f46d26aca130f7430d1"
server_key_version: "encryptionkey@0"
I20260925 09:14:55.262543 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "b82953dea67d413c94ef533426d8ad08"
format_stamp: "Formatted at 2026-09-25 09:14:55 on dist-test-slave-l6qh"
server_key: "3b2b77688be618bc281506b30c05b58b"
server_key_iv: "c43908f051192f46d26aca130f7430d1"
server_key_version: "encryptionkey@0"
I20260925 09:14:55.267968 12610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:14:55.271566 12624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:55.273355 12610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:55.273540 12610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "b82953dea67d413c94ef533426d8ad08"
format_stamp: "Formatted at 2026-09-25 09:14:55 on dist-test-slave-l6qh"
server_key: "3b2b77688be618bc281506b30c05b58b"
server_key_iv: "c43908f051192f46d26aca130f7430d1"
server_key_version: "encryptionkey@0"
I20260925 09:14:55.273840 12610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:55.288965 12610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:55.289973 12610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:55.290282 12610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:55.291298 12610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:55.292974 12610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:55.293102 12610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:55.293264 12610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:55.293365 12610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 09:14:55.327473 12610 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:35793
I20260925 09:14:55.327533 12736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:35793 every 8 connection(s)
I20260925 09:14:55.328827 12610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:14:55.335171 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 12610
I20260925 09:14:55.335386 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:14:55.335997 3149 external_mini_cluster.cc:1428] Setting key 11015d42a1cc3296023f2c99262f9fa1
I20260925 09:14:55.339023 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:36383
--builtin_ntp_servers=127.3.19.84:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 09:14:55.342509 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:55.331470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:54119 (local address 127.3.19.126:36383)
0925 09:14:55.331922 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:55.331927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:55.333068 (+ 1141us) server_negotiation.cc:407] Connection header received
0925 09:14:55.334877 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:55.334883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:55.334972 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:55.335091 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:55.336249 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:55.337201 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:55.339517 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:55.339917 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:55.340018 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:55.341765 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:55.341770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:55.341773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:55.341793 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:55.341833 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:55.341845 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:55.341922 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:55.342176 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 09:14:55.342256 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":261,"threads_started":1}
I20260925 09:14:55.344120 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:55.331989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:36383 (local address 127.3.19.65:54119)
0925 09:14:55.332903 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:55.332944 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:55.334609 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:55.335335 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:55.335346 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:55.335447 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:55.336063 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:55.336080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:55.337358 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:55.337363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:55.339360 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:55.339379 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:55.340162 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:55.340173 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:55.340536 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:55.341386 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:55.341392 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:55.341451 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:55.341558 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:55.341943 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:55.341955 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:55.341966 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 09:14:55.342165 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 09:14:55.342622 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":280,"threads_started":1}
I20260925 09:14:55.345795 12737 heartbeater.cc:347] Connected to a master server at 127.3.19.126:36383
I20260925 09:14:55.346119 12737 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:55.346948 12737 heartbeater.cc:511] Master 127.3.19.126:36383 requested a full tablet report, sending...
I20260925 09:14:55.349182 12551 ts_manager.cc:194] Registered new tserver with Master: b82953dea67d413c94ef533426d8ad08 (127.3.19.65:35793)
I20260925 09:14:55.350756 12551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.19.65:54119
W20260925 09:14:55.691643 12741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:55.692021 12741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:55.692178 12741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:55.699962 12741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:55.700146 12741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:55.700346 12741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:14:55.710387 12741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:36383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:55.712028 12741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:55.714082 12741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:55.728267 12746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:55.728308 12747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:55.728349 12749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:55.729476 12741 server_base.cc:1034] running on GCE node
I20260925 09:14:55.730232 12741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:55.731227 12741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:55.732455 12741 hybrid_clock.cc:648] HybridClock initialized: now 1790327695732398 us; error 75 us; skew 500 ppm
I20260925 09:14:55.732787 12741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:55.735167 12741 webserver.cc:460] Webserver started at http://127.3.19.66:45181/ using document root <none> and password file <none>
I20260925 09:14:55.735895 12741 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:55.736011 12741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:55.736339 12741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:55.739425 12741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "efece266a685419eb7ee1a69bc2cf1b3"
format_stamp: "Formatted at 2026-09-25 09:14:55 on dist-test-slave-l6qh"
server_key: "a1247a686a1603860a1c0545d8b00efc"
server_key_iv: "3a0d62dc3dc60a85500b1f4e5320c1f6"
server_key_version: "encryptionkey@0"
I20260925 09:14:55.740269 12741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "efece266a685419eb7ee1a69bc2cf1b3"
format_stamp: "Formatted at 2026-09-25 09:14:55 on dist-test-slave-l6qh"
server_key: "a1247a686a1603860a1c0545d8b00efc"
server_key_iv: "3a0d62dc3dc60a85500b1f4e5320c1f6"
server_key_version: "encryptionkey@0"
I20260925 09:14:55.745731 12741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 09:14:55.749221 12755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:55.750759 12741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:14:55.750952 12741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "efece266a685419eb7ee1a69bc2cf1b3"
format_stamp: "Formatted at 2026-09-25 09:14:55 on dist-test-slave-l6qh"
server_key: "a1247a686a1603860a1c0545d8b00efc"
server_key_iv: "3a0d62dc3dc60a85500b1f4e5320c1f6"
server_key_version: "encryptionkey@0"
I20260925 09:14:55.751199 12741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:55.780707 12741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:55.781688 12741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:55.781966 12741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:55.782959 12741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:55.784444 12741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:55.784530 12741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:55.784686 12741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:55.784771 12741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:55.818773 12741 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:40217
I20260925 09:14:55.818861 12869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:40217 every 8 connection(s)
I20260925 09:14:55.820114 12741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:14:55.826501 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 12741
I20260925 09:14:55.826678 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:14:55.827235 3149 external_mini_cluster.cc:1428] Setting key 8b0e5042403c29ac20362f6ff29a24d6
I20260925 09:14:55.830482 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:36383
--builtin_ntp_servers=127.3.19.84:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 09:14:55.834501 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:55.822829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:41205 (local address 127.3.19.126:36383)
0925 09:14:55.823002 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:55.823008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:55.824383 (+ 1375us) server_negotiation.cc:407] Connection header received
0925 09:14:55.826344 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:55.826349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:55.826418 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:55.826507 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:55.828133 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:55.828927 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:55.831355 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:55.831759 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:55.831870 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:55.833758 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:55.833762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:55.833765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:55.833786 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:55.833816 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:55.833829 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:55.833906 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:55.834171 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 09:14:55.834254 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 09:14:55.836097 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:55.823243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:36383 (local address 127.3.19.66:41205)
0925 09:14:55.824223 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:55.824259 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:55.826029 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:55.826799 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:55.826810 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:55.826944 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:55.827892 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:55.827924 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:55.829085 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:55.829092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:55.831194 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:55.831214 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:55.832007 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:55.832016 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:55.832361 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:55.833355 (+ 994us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:55.833361 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:55.833418 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:55.833533 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:55.833927 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:55.833939 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:55.833950 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 09:14:55.834146 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 09:14:55.834608 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":292,"threads_started":1}
I20260925 09:14:55.837695 12870 heartbeater.cc:347] Connected to a master server at 127.3.19.126:36383
I20260925 09:14:55.838014 12870 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:55.838816 12870 heartbeater.cc:511] Master 127.3.19.126:36383 requested a full tablet report, sending...
I20260925 09:14:55.840243 12551 ts_manager.cc:194] Registered new tserver with Master: efece266a685419eb7ee1a69bc2cf1b3 (127.3.19.66:40217)
I20260925 09:14:55.841064 12551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.19.66:41205
W20260925 09:14:56.166846 12873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:56.167143 12873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:56.167297 12873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:56.174705 12873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:56.174824 12873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:56.174993 12873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:14:56.184391 12873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:36383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:56.185808 12873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:56.187635 12873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:56.201098 12878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:56.201056 12881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:56.201144 12879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:56.202432 12873 server_base.cc:1034] running on GCE node
I20260925 09:14:56.203166 12873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:56.204201 12873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:56.205427 12873 hybrid_clock.cc:648] HybridClock initialized: now 1790327696205360 us; error 70 us; skew 500 ppm
I20260925 09:14:56.205765 12873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:56.208303 12873 webserver.cc:460] Webserver started at http://127.3.19.67:33927/ using document root <none> and password file <none>
I20260925 09:14:56.209060 12873 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:56.209187 12873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:56.209574 12873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:56.212661 12873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "abc1746867b94de794fefbf15de9f9f4"
format_stamp: "Formatted at 2026-09-25 09:14:56 on dist-test-slave-l6qh"
server_key: "65b411c1e675f5a5ec9d4a483f806e61"
server_key_iv: "9b37f7d8c24bd1522dfd947caf79dfdd"
server_key_version: "encryptionkey@0"
I20260925 09:14:56.213541 12873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "abc1746867b94de794fefbf15de9f9f4"
format_stamp: "Formatted at 2026-09-25 09:14:56 on dist-test-slave-l6qh"
server_key: "65b411c1e675f5a5ec9d4a483f806e61"
server_key_iv: "9b37f7d8c24bd1522dfd947caf79dfdd"
server_key_version: "encryptionkey@0"
I20260925 09:14:56.219178 12873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:14:56.222875 12887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:56.224270 12873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:14:56.224462 12873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "abc1746867b94de794fefbf15de9f9f4"
format_stamp: "Formatted at 2026-09-25 09:14:56 on dist-test-slave-l6qh"
server_key: "65b411c1e675f5a5ec9d4a483f806e61"
server_key_iv: "9b37f7d8c24bd1522dfd947caf79dfdd"
server_key_version: "encryptionkey@0"
I20260925 09:14:56.224716 12873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:56.239320 12873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:56.240209 12873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:56.240485 12873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:56.241434 12873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:56.242914 12873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:56.243031 12873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:56.243162 12873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:56.243247 12873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:56.276749 12873 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:32819
I20260925 09:14:56.276818 13000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:32819 every 8 connection(s)
I20260925 09:14:56.278303 12873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:14:56.287694 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 12873
I20260925 09:14:56.287899 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:14:56.288362 3149 external_mini_cluster.cc:1428] Setting key 4f9e3bebcc5fdf8fc6b7606215aa444b
I20260925 09:14:56.292681 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:56.281150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:36157 (local address 127.3.19.126:36383)
0925 09:14:56.281423 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:56.281429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:56.282998 (+ 1569us) server_negotiation.cc:407] Connection header received
0925 09:14:56.284963 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:56.284968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:56.285072 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:56.285189 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:56.286423 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:56.287320 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:56.289353 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:56.290059 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:56.290217 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:56.291952 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:56.291957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:56.291961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:56.291979 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:56.292013 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:56.292026 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:56.292105 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:56.292315 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 09:14:56.292396 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 09:14:56.293761 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:56.281805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:36383 (local address 127.3.19.67:36157)
0925 09:14:56.282817 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:56.282868 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:56.284651 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:56.285478 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:56.285487 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:56.285599 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:56.286239 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:56.286257 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:56.287513 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:56.287518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:56.289177 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:14:56.289191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:14:56.290308 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:14:56.290316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:14:56.290639 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:56.291453 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:56.291457 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:56.291514 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:56.291624 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:56.292114 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:56.292122 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:56.292128 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 09:14:56.292274 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 09:14:56.292595 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":297,"threads_started":1}
I20260925 09:14:56.295146 13001 heartbeater.cc:347] Connected to a master server at 127.3.19.126:36383
I20260925 09:14:56.295483 13001 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:56.296335 13001 heartbeater.cc:511] Master 127.3.19.126:36383 requested a full tablet report, sending...
I20260925 09:14:56.297816 12551 ts_manager.cc:194] Registered new tserver with Master: abc1746867b94de794fefbf15de9f9f4 (127.3.19.67:32819)
I20260925 09:14:56.298601 12551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.19.67:36157
I20260925 09:14:56.304150 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 09:14:56.310850 12739 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:56.309196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.3.19.126:36383)
0925 09:14:56.309494 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:56.309503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:56.309804 (+ 301us) server_negotiation.cc:407] Connection header received
0925 09:14:56.309972 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:56.309976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:56.310068 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:56.310158 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:56.310608 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47486: BlockingRecv error: recv got EOF from 127.0.0.1:47486 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 09:14:56.311416 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 12610
I20260925 09:14:56.319514 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:56.312471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60202 (local address 127.3.19.65:35793)
0925 09:14:56.313018 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:56.313027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:56.313085 (+ 58us) server_negotiation.cc:407] Connection header received
0925 09:14:56.313144 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:56.313150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:56.313422 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:56.313557 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:56.313881 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:56.314839 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:56.316199 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:56.316950 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:56.317064 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:56.318168 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:56.318176 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:56.318190 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:56.318788 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:56.318841 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:56.318860 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:56.318951 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:56.319067 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 09:14:56.319198 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":302,"threads_started":1}
I20260925 09:14:56.320868 12732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:56.401262 12737 heartbeater.cc:503] Master 127.3.19.126:36383 was elected leader, sending a full tablet report...
I20260925 09:14:56.402431 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 12610
W20260925 09:14:56.420020 12537 connection.cc:535] server connection from 127.3.19.65:54119 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 09:14:56.420331 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 12741
I20260925 09:14:56.428145 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:56.421192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.3.19.66:40217)
0925 09:14:56.421788 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:56.421796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:56.421823 (+ 27us) server_negotiation.cc:407] Connection header received
0925 09:14:56.421888 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:56.421894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:56.422110 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:56.422241 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:56.422632 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:56.423611 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:56.424870 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:56.425653 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:56.425765 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:56.426773 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:56.426781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:56.426797 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:56.427399 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:56.427447 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:56.427466 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:56.427576 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:56.427733 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 09:14:56.427846 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":288,"threads_started":1}
I20260925 09:14:56.429523 12865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:56.509447 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 12741
I20260925 09:14:56.526588 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 12873
I20260925 09:14:56.535493 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:56.527652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45694 (local address 127.3.19.67:32819)
0925 09:14:56.528234 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:56.528242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:56.528271 (+ 29us) server_negotiation.cc:407] Connection header received
0925 09:14:56.528334 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:56.528341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:56.528587 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:56.528846 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:56.529337 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:56.530522 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:56.531854 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:14:56.532453 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:14:56.532596 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:14:56.533949 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:56.533958 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:56.533972 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:56.534604 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:56.534655 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:56.534680 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:56.534817 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:56.534969 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 09:14:56.535123 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":293,"threads_started":1}
I20260925 09:14:56.536854 12996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:56.619227 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 12873
I20260925 09:14:56.635305 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 12520
I20260925 09:14:56.636545 12581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:56.737936 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 12520
2026-09-25T09:14:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2492 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T09:14:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:14:56Z Disabled control of system clock
I20260925 09:14:56.813897 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:34631
--webserver_interface=127.3.19.126
--webserver_port=0
--builtin_ntp_servers=127.3.19.84:44527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:34631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 09:14:57.156322 13020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:57.156639 13020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:57.156731 13020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:57.164177 13020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:14:57.164311 13020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:57.164350 13020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:57.164403 13020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:14:57.164449 13020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:14:57.173985 13020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:34631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:34631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:57.175311 13020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:57.177205 13020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:57.189487 13028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:57.189458 13025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:57.189412 13026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:57.190310 13020 server_base.cc:1034] running on GCE node
I20260925 09:14:57.191172 13020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:57.192667 13020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:57.193876 13020 hybrid_clock.cc:648] HybridClock initialized: now 1790327697193833 us; error 34 us; skew 500 ppm
I20260925 09:14:57.194205 13020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:57.196758 13020 webserver.cc:460] Webserver started at http://127.3.19.126:36115/ using document root <none> and password file <none>
I20260925 09:14:57.197599 13020 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:57.197698 13020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:57.197983 13020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:57.201067 13020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "64c9f585a7d349dcad463c3c9eda61b0"
format_stamp: "Formatted at 2026-09-25 09:14:57 on dist-test-slave-l6qh"
server_key: "9435ca849cb9cfefcba77e73973baccc"
server_key_iv: "c82eb18d1e8377e43f6229d77b6c09c0"
server_key_version: "encryptionkey@0"
I20260925 09:14:57.202001 13020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "64c9f585a7d349dcad463c3c9eda61b0"
format_stamp: "Formatted at 2026-09-25 09:14:57 on dist-test-slave-l6qh"
server_key: "9435ca849cb9cfefcba77e73973baccc"
server_key_iv: "c82eb18d1e8377e43f6229d77b6c09c0"
server_key_version: "encryptionkey@0"
I20260925 09:14:57.207647 13020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:14:57.211422 13034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:57.212945 13020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:14:57.213100 13020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "64c9f585a7d349dcad463c3c9eda61b0"
format_stamp: "Formatted at 2026-09-25 09:14:57 on dist-test-slave-l6qh"
server_key: "9435ca849cb9cfefcba77e73973baccc"
server_key_iv: "c82eb18d1e8377e43f6229d77b6c09c0"
server_key_version: "encryptionkey@0"
I20260925 09:14:57.213402 13020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:57.219728 13020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:57.220582 13020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:57.220857 13020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:57.240391 13020 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:34631
I20260925 09:14:57.240396 13086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:34631 every 8 connection(s)
I20260925 09:14:57.241968 13020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:14:57.246220 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:14:57.248297 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 13020
I20260925 09:14:57.248525 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:14:57.249399 3149 external_mini_cluster.cc:1428] Setting key be1fe0aeb693e5c5e18d5459bd1186e6
I20260925 09:14:57.259155 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:57.251788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.3.19.126:34631)
0925 09:14:57.253062 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:57.253075 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:57.253121 (+ 46us) server_negotiation.cc:407] Connection header received
0925 09:14:57.254885 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:57.254909 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:57.255367 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:57.255813 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:57.257186 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:57.257197 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:57.257212 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:57.257284 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:57.257362 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:57.257386 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:57.257647 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:57.257791 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 09:14:57.258083 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":548,"threads_started":1}
I20260925 09:14:57.259614 13087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0: Bootstrap starting.
I20260925 09:14:57.262984 13087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0: Neither blocks nor log segments found. Creating new log.
I20260925 09:14:57.264106 13087 log.cc:824] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0: Log is configured to *not* fsync() on all Append() calls
I20260925 09:14:57.267467 13087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0: No bootstrap required, opened a new log
I20260925 09:14:57.275785 13087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c9f585a7d349dcad463c3c9eda61b0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34631 } }
I20260925 09:14:57.276258 13087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:14:57.276408 13087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64c9f585a7d349dcad463c3c9eda61b0, State: Initialized, Role: FOLLOWER
I20260925 09:14:57.277190 13087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c9f585a7d349dcad463c3c9eda61b0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34631 } }
I20260925 09:14:57.277487 13087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:14:57.277631 13087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:14:57.277837 13087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:14:57.280148 13087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c9f585a7d349dcad463c3c9eda61b0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34631 } }
I20260925 09:14:57.280771 13087 leader_election.cc:302] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64c9f585a7d349dcad463c3c9eda61b0; no voters:
I20260925 09:14:57.281528 13087 leader_election.cc:288] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:14:57.281723 13092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:14:57.282652 13092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [term 1 LEADER]: Becoming Leader. State: Replica: 64c9f585a7d349dcad463c3c9eda61b0, State: Running, Role: LEADER
I20260925 09:14:57.283291 13092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c9f585a7d349dcad463c3c9eda61b0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34631 } }
I20260925 09:14:57.283919 13087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:14:57.286773 13093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64c9f585a7d349dcad463c3c9eda61b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c9f585a7d349dcad463c3c9eda61b0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34631 } } }
I20260925 09:14:57.286916 13094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64c9f585a7d349dcad463c3c9eda61b0. Latest consensus state: current_term: 1 leader_uuid: "64c9f585a7d349dcad463c3c9eda61b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c9f585a7d349dcad463c3c9eda61b0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34631 } } }
I20260925 09:14:57.287369 13093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:57.287379 13094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0 [sys.catalog]: This master's current role is: LEADER
I20260925 09:14:57.290444 13097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:14:57.295931 13097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:14:57.306322 13097 catalog_manager.cc:1347] Generated new cluster ID: 340b9bb1ece14692b25cc61c8bcd9edf
I20260925 09:14:57.306456 13097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:14:57.361109 13097 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:14:57.361604 13097 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:14:57.370797 13097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64c9f585a7d349dcad463c3c9eda61b0: Generated new TSK 0
I20260925 09:14:57.371670 13097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:14:57.378336 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:34631
--builtin_ntp_servers=127.3.19.84:44527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 09:14:57.727262 13111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:57.727710 13111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:57.727902 13111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:57.735535 13111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:57.735654 13111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:57.735841 13111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:14:57.745492 13111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:34631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:57.746846 13111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:57.748651 13111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:57.762379 13119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:57.762458 13116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:57.762436 13117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:57.763513 13111 server_base.cc:1034] running on GCE node
I20260925 09:14:57.764236 13111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:57.765213 13111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:57.766557 13111 hybrid_clock.cc:648] HybridClock initialized: now 1790327697766494 us; error 61 us; skew 500 ppm
I20260925 09:14:57.766924 13111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:57.770257 13111 webserver.cc:460] Webserver started at http://127.3.19.65:36955/ using document root <none> and password file <none>
I20260925 09:14:57.771157 13111 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:57.771256 13111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:57.771601 13111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:57.776554 13111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "1c90bb3b80664607bc35247243c71598"
format_stamp: "Formatted at 2026-09-25 09:14:57 on dist-test-slave-l6qh"
server_key: "fee725e58d6234bda77a32c2e760b7e9"
server_key_iv: "e8972304732a0d3397af233eff131aaf"
server_key_version: "encryptionkey@0"
I20260925 09:14:57.777580 13111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "1c90bb3b80664607bc35247243c71598"
format_stamp: "Formatted at 2026-09-25 09:14:57 on dist-test-slave-l6qh"
server_key: "fee725e58d6234bda77a32c2e760b7e9"
server_key_iv: "e8972304732a0d3397af233eff131aaf"
server_key_version: "encryptionkey@0"
I20260925 09:14:57.783399 13111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260925 09:14:57.787454 13125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:57.789276 13111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:14:57.789461 13111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "1c90bb3b80664607bc35247243c71598"
format_stamp: "Formatted at 2026-09-25 09:14:57 on dist-test-slave-l6qh"
server_key: "fee725e58d6234bda77a32c2e760b7e9"
server_key_iv: "e8972304732a0d3397af233eff131aaf"
server_key_version: "encryptionkey@0"
I20260925 09:14:57.789695 13111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:57.795683 13111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:57.796517 13111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:57.796792 13111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:57.797783 13111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:57.799428 13111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:57.799523 13111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:57.799659 13111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:57.799737 13111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:57.831084 13111 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:37665
I20260925 09:14:57.831138 13238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:37665 every 8 connection(s)
I20260925 09:14:57.832463 13111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:14:57.835719 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 13111
I20260925 09:14:57.835933 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:14:57.836556 3149 external_mini_cluster.cc:1428] Setting key d4cd0fcfa7481e978d5018e8cd4a9dc3
I20260925 09:14:57.839529 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:34631
--builtin_ntp_servers=127.3.19.84:44527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 09:14:57.841428 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:57.835212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:41803 (local address 127.3.19.126:34631)
0925 09:14:57.835698 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:57.835713 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:57.837265 (+ 1552us) server_negotiation.cc:407] Connection header received
0925 09:14:57.838945 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:57.838950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:57.839013 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:57.839138 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:57.840527 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:57.840535 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:57.840543 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:57.840569 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:57.840636 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:57.840656 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:57.840793 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:57.840966 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 09:14:57.841032 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":231,"threads_started":1}
I20260925 09:14:57.842255 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:57.835909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:34631 (local address 127.3.19.65:41803)
0925 09:14:57.837049 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:57.837103 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:57.838671 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:57.839344 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:57.839352 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:57.839448 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:57.840224 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:57.840226 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:57.840267 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:57.840352 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:57.840774 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:57.840781 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:57.840786 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 09:14:57.840996 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 09:14:57.841203 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":352,"threads_started":1}
I20260925 09:14:57.843860 13239 heartbeater.cc:347] Connected to a master server at 127.3.19.126:34631
I20260925 09:14:57.844167 13239 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:57.844775 13239 heartbeater.cc:511] Master 127.3.19.126:34631 requested a full tablet report, sending...
I20260925 09:14:57.847087 13051 ts_manager.cc:194] Registered new tserver with Master: 1c90bb3b80664607bc35247243c71598 (127.3.19.65:37665)
W20260925 09:14:58.174000 13243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:58.174299 13243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:58.174464 13243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:58.181800 13243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:58.181913 13243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:58.182086 13243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:14:58.191563 13243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:34631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:58.193004 13243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:58.194898 13243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:58.208228 13248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:58.208289 13251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:58.208330 13249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:58.209007 13243 server_base.cc:1034] running on GCE node
I20260925 09:14:58.209687 13243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:58.210676 13243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:58.211902 13243 hybrid_clock.cc:648] HybridClock initialized: now 1790327698211855 us; error 43 us; skew 500 ppm
I20260925 09:14:58.212227 13243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:58.214512 13243 webserver.cc:460] Webserver started at http://127.3.19.66:41975/ using document root <none> and password file <none>
I20260925 09:14:58.215199 13243 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:58.215315 13243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:58.215641 13243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:58.218720 13243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "4f69125a475641b78f9551b95d762e56"
format_stamp: "Formatted at 2026-09-25 09:14:58 on dist-test-slave-l6qh"
server_key: "90811d816379da987eec53fd341e3e3b"
server_key_iv: "44bd6947b7befa6b2ad144efafa32f4d"
server_key_version: "encryptionkey@0"
I20260925 09:14:58.219558 13243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "4f69125a475641b78f9551b95d762e56"
format_stamp: "Formatted at 2026-09-25 09:14:58 on dist-test-slave-l6qh"
server_key: "90811d816379da987eec53fd341e3e3b"
server_key_iv: "44bd6947b7befa6b2ad144efafa32f4d"
server_key_version: "encryptionkey@0"
I20260925 09:14:58.225271 13243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 09:14:58.228780 13257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:58.230234 13243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:58.230415 13243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "4f69125a475641b78f9551b95d762e56"
format_stamp: "Formatted at 2026-09-25 09:14:58 on dist-test-slave-l6qh"
server_key: "90811d816379da987eec53fd341e3e3b"
server_key_iv: "44bd6947b7befa6b2ad144efafa32f4d"
server_key_version: "encryptionkey@0"
I20260925 09:14:58.230636 13243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:58.236639 13243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:58.237598 13243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:58.237866 13243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:58.238834 13243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:58.240371 13243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:58.240456 13243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:58.240595 13243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:58.240679 13243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:58.276518 13243 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:40233
I20260925 09:14:58.276609 13370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:40233 every 8 connection(s)
I20260925 09:14:58.277971 13243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:14:58.285993 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 13243
I20260925 09:14:58.286216 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:14:58.286659 3149 external_mini_cluster.cc:1428] Setting key baab37ab4953f0b254c679d71e341411
I20260925 09:14:58.289705 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:34631
--builtin_ntp_servers=127.3.19.84:44527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 09:14:58.293808 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:58.280913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:46145 (local address 127.3.19.126:34631)
0925 09:14:58.281176 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:58.281183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:58.282895 (+ 1712us) server_negotiation.cc:407] Connection header received
0925 09:14:58.284739 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:58.284745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:58.284825 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:58.284933 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:58.292722 (+ 7789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:58.292730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:58.292738 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:58.292766 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:58.292813 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:58.292832 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:58.292974 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:58.293287 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 09:14:58.293390 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 09:14:58.295048 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:58.281615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:34631 (local address 127.3.19.66:46145)
0925 09:14:58.282674 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:58.282715 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:58.284415 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:58.285245 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:58.285282 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:58.285393 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:58.292287 (+ 6894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:58.292291 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:58.292387 (+ 96us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:58.292508 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:58.292979 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:58.292990 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:58.292999 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 09:14:58.293494 (+ 495us) client_negotiation.cc:240] Negotiation successful
0925 09:14:58.293832 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":325,"threads_started":1}
I20260925 09:14:58.296595 13371 heartbeater.cc:347] Connected to a master server at 127.3.19.126:34631
I20260925 09:14:58.296914 13371 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:58.297567 13371 heartbeater.cc:511] Master 127.3.19.126:34631 requested a full tablet report, sending...
I20260925 09:14:58.299141 13051 ts_manager.cc:194] Registered new tserver with Master: 4f69125a475641b78f9551b95d762e56 (127.3.19.66:40233)
W20260925 09:14:58.639068 13374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:14:58.639391 13374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:14:58.639537 13374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:14:58.647728 13374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:14:58.647845 13374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:14:58.648059 13374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:14:58.657707 13374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:34631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:14:58.659034 13374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:14:58.660868 13374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:14:58.674712 13379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:58.674957 13382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:14:58.674983 13380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:14:58.675758 13374 server_base.cc:1034] running on GCE node
I20260925 09:14:58.676478 13374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:14:58.677517 13374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:14:58.678714 13374 hybrid_clock.cc:648] HybridClock initialized: now 1790327698678667 us; error 44 us; skew 500 ppm
I20260925 09:14:58.679042 13374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:14:58.681466 13374 webserver.cc:460] Webserver started at http://127.3.19.67:40359/ using document root <none> and password file <none>
I20260925 09:14:58.682188 13374 fs_manager.cc:359] Metadata directory not provided
I20260925 09:14:58.682299 13374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:14:58.682626 13374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:14:58.685719 13374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "01e3711d04004a7894e1f127c16910ae"
format_stamp: "Formatted at 2026-09-25 09:14:58 on dist-test-slave-l6qh"
server_key: "eedf082d460c950cb0350bc51c640bac"
server_key_iv: "7e8232cf4946394401c165cec6d76821"
server_key_version: "encryptionkey@0"
I20260925 09:14:58.686523 13374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "01e3711d04004a7894e1f127c16910ae"
format_stamp: "Formatted at 2026-09-25 09:14:58 on dist-test-slave-l6qh"
server_key: "eedf082d460c950cb0350bc51c640bac"
server_key_iv: "7e8232cf4946394401c165cec6d76821"
server_key_version: "encryptionkey@0"
I20260925 09:14:58.692128 13374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 09:14:58.695914 13388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:58.697484 13374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:14:58.697659 13374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "01e3711d04004a7894e1f127c16910ae"
format_stamp: "Formatted at 2026-09-25 09:14:58 on dist-test-slave-l6qh"
server_key: "eedf082d460c950cb0350bc51c640bac"
server_key_iv: "7e8232cf4946394401c165cec6d76821"
server_key_version: "encryptionkey@0"
I20260925 09:14:58.697889 13374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:14:58.703742 13374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:14:58.704574 13374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:14:58.704847 13374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:14:58.705893 13374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:14:58.707346 13374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:14:58.707432 13374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:58.707578 13374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:14:58.707649 13374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:14:58.740286 13374 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:39863
I20260925 09:14:58.740373 13501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:39863 every 8 connection(s)
I20260925 09:14:58.741626 13374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:14:58.746764 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 13374
I20260925 09:14:58.746955 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:14:58.747398 3149 external_mini_cluster.cc:1428] Setting key c4f522076c26bf269a1f21ef364e2186
I20260925 09:14:58.750464 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:58.744436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:51463 (local address 127.3.19.126:34631)
0925 09:14:58.744633 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:58.744638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:58.746174 (+ 1536us) server_negotiation.cc:407] Connection header received
0925 09:14:58.748032 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:58.748037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:58.748128 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:58.748238 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:58.749740 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:58.749744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:58.749747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:58.749763 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:58.749793 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:58.749804 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:58.749892 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:58.750120 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 09:14:58.750186 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 09:14:58.751456 13504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:58.744984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:34631 (local address 127.3.19.67:51463)
0925 09:14:58.746013 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0925 09:14:58.746054 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 09:14:58.747721 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:14:58.748500 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:14:58.748512 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:14:58.748665 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:14:58.749390 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:14:58.749393 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:14:58.749459 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:14:58.749581 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:14:58.749898 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:14:58.749907 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:14:58.749914 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 09:14:58.750204 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 09:14:58.750487 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":355,"threads_started":1}
I20260925 09:14:58.752707 13502 heartbeater.cc:347] Connected to a master server at 127.3.19.126:34631
I20260925 09:14:58.753006 13502 heartbeater.cc:464] Registering TS with master...
I20260925 09:14:58.753641 13502 heartbeater.cc:511] Master 127.3.19.126:34631 requested a full tablet report, sending...
I20260925 09:14:58.755163 13051 ts_manager.cc:194] Registered new tserver with Master: 01e3711d04004a7894e1f127c16910ae (127.3.19.67:39863)
I20260925 09:14:58.763206 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 09:14:58.769553 13241 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:14:58.767969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.3.19.126:34631)
0925 09:14:58.768171 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:58.768177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:58.768626 (+ 449us) server_negotiation.cc:407] Connection header received
0925 09:14:58.768757 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:58.768762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:58.768823 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:58.768934 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:58.769317 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42264: BlockingRecv error: recv got EOF from 127.0.0.1:42264 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 09:14:58.769968 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 13111
I20260925 09:14:58.775607 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:58.771404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.3.19.65:37665)
0925 09:14:58.772834 (+ 1430us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:58.772847 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:58.772885 (+ 38us) server_negotiation.cc:407] Connection header received
0925 09:14:58.772971 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:58.772981 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:58.773332 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:58.773543 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:58.774810 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:58.774820 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:58.774837 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:58.774869 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:58.774919 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:58.774941 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:58.775094 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:58.775186 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 09:14:58.775297 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1243,"thread_start_us":1110,"threads_started":1}
I20260925 09:14:58.777019 13234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:58.856958 13239 heartbeater.cc:503] Master 127.3.19.126:34631 was elected leader, sending a full tablet report...
I20260925 09:14:58.858639 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 13111
I20260925 09:14:58.875752 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 13243
I20260925 09:14:58.879748 13513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:58.876723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.3.19.66:40233)
0925 09:14:58.877218 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:58.877279 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:58.877315 (+ 36us) server_negotiation.cc:407] Connection header received
0925 09:14:58.877379 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:58.877385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:58.877608 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:58.877769 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:58.878800 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:58.878807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:58.878820 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:58.878834 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:58.878883 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:58.878901 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:58.879018 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:58.879334 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 09:14:58.879476 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":242,"threads_started":1}
I20260925 09:14:58.880975 13366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:58.964257 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 13243
I20260925 09:14:58.980710 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 13374
I20260925 09:14:58.985018 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:14:58.981774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43676 (local address 127.3.19.67:39863)
0925 09:14:58.982409 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0925 09:14:58.982417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:14:58.982448 (+ 31us) server_negotiation.cc:407] Connection header received
0925 09:14:58.982511 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:14:58.982517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:14:58.982752 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:14:58.982930 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:14:58.984190 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:14:58.984197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:14:58.984210 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:14:58.984226 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:14:58.984267 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:14:58.984281 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:14:58.984475 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 09:14:58.984561 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 09:14:58.984670 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":327,"threads_started":1}
I20260925 09:14:58.986600 13497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:59.068756 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 13374
I20260925 09:14:59.085680 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 13020
I20260925 09:14:59.087092 13082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:14:59.188405 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 13020
2026-09-25T09:14:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2451 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:14:59 dist-test-slave-l6qh krb5kdc[13520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:14:59 dist-test-slave-l6qh krb5kdc[13520](info): set up 2 sockets
Sep 25 09:14:59 dist-test-slave-l6qh krb5kdc[13520](info): commencing operation
krb5kdc: starting...
W20260925 09:15:00.308856 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:15:00 dist-test-slave-l6qh krb5kdc[13520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:15:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:15:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:15:00.573357 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33795
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:33795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:15:00.921418 13533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:15:00.921722 13533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:15:00.921886 13533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:15:00.929922 13533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:15:00.930070 13533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:15:00.930152 13533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:15:00.930230 13533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:15:00.930291 13533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:15:00.940347 13533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:33795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:15:00.941818 13533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:15:00.943158 13533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:15:00.955087 13539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:00.955171 13538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:00.955147 13541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:15:00.956619 13533 server_base.cc:1034] running on GCE node
I20260925 09:15:00.957567 13533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:15:00.959046 13533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:15:00.960290 13533 hybrid_clock.cc:648] HybridClock initialized: now 1790327700960246 us; error 60 us; skew 500 ppm
I20260925 09:15:00.960652 13533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:15:00 dist-test-slave-l6qh krb5kdc[13520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:15:00.963886 13533 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:15:00.965637 13533 webserver.cc:460] Webserver started at http://127.3.19.126:41797/ using document root <none> and password file <none>
I20260925 09:15:00.966390 13533 fs_manager.cc:359] Metadata directory not provided
I20260925 09:15:00.966547 13533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:15:00.966879 13533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:15:00.970037 13533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "fbc30c7beaa041f0badbb147e304e138"
format_stamp: "Formatted at 2026-09-25 09:15:00 on dist-test-slave-l6qh"
server_key: "24da7f656e20abd3134a05ea26f6e228"
server_key_iv: "eacb4b1a7729a18077b1bdd508f54348"
server_key_version: "encryptionkey@0"
I20260925 09:15:00.970824 13533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "fbc30c7beaa041f0badbb147e304e138"
format_stamp: "Formatted at 2026-09-25 09:15:00 on dist-test-slave-l6qh"
server_key: "24da7f656e20abd3134a05ea26f6e228"
server_key_iv: "eacb4b1a7729a18077b1bdd508f54348"
server_key_version: "encryptionkey@0"
I20260925 09:15:00.976176 13533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:15:00.979918 13548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:00.981452 13533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:15:00.981671 13533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "fbc30c7beaa041f0badbb147e304e138"
format_stamp: "Formatted at 2026-09-25 09:15:00 on dist-test-slave-l6qh"
server_key: "24da7f656e20abd3134a05ea26f6e228"
server_key_iv: "eacb4b1a7729a18077b1bdd508f54348"
server_key_version: "encryptionkey@0"
I20260925 09:15:00.981890 13533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:15:00.994836 13533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:15:00.999291 13533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:15:00.999693 13533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:15:01.020254 13533 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:33795
I20260925 09:15:01.020277 13600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:33795 every 8 connection(s)
I20260925 09:15:01.021929 13533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:15:01.025695 13601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:15:01.029367 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 13533
I20260925 09:15:01.029618 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:15:01.030423 3149 external_mini_cluster.cc:1428] Setting key 0ef0554f440a81f939602fc00cdcc802
I20260925 09:15:01.039103 13601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138: Bootstrap starting.
I20260925 09:15:01.042575 13601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138: Neither blocks nor log segments found. Creating new log.
I20260925 09:15:01.043681 13601 log.cc:824] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138: Log is configured to *not* fsync() on all Append() calls
Sep 25 09:15:01 dist-test-slave-l6qh krb5kdc[13520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:15:01.047277 13601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138: No bootstrap required, opened a new log
I20260925 09:15:01.055745 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:01.032650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.3.19.126:33795)
0925 09:15:01.033536 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:01.033581 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:01.033625 (+ 44us) server_negotiation.cc:407] Connection header received
0925 09:15:01.035347 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:01.035371 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:01.035858 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:01.036347 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:01.038113 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:01.039494 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:01.040808 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:01.041589 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:01.041795 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:01.046312 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:01.046326 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:01.046355 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:01.046406 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:01.050653 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:01.051453 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:01.051467 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:01.051478 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:01.051698 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:01.052140 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:01.052146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:01.052149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:01.052928 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:01.053305 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:01.053730 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 09:15:01.054261 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":413,"threads_started":1}
I20260925 09:15:01.058202 13601 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc30c7beaa041f0badbb147e304e138" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33795 } }
I20260925 09:15:01.058569 13601 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:15:01.058729 13601 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc30c7beaa041f0badbb147e304e138, State: Initialized, Role: FOLLOWER
I20260925 09:15:01.059384 13601 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc30c7beaa041f0badbb147e304e138" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33795 } }
I20260925 09:15:01.059604 13601 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:15:01.059702 13601 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:15:01.059821 13601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:15:01.062304 13601 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc30c7beaa041f0badbb147e304e138" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33795 } }
I20260925 09:15:01.062784 13601 leader_election.cc:302] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbc30c7beaa041f0badbb147e304e138; no voters:
I20260925 09:15:01.063335 13601 leader_election.cc:288] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:15:01.063495 13606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:15:01.064699 13606 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [term 1 LEADER]: Becoming Leader. State: Replica: fbc30c7beaa041f0badbb147e304e138, State: Running, Role: LEADER
I20260925 09:15:01.065315 13606 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc30c7beaa041f0badbb147e304e138" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33795 } }
I20260925 09:15:01.065639 13601 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:15:01.068159 13608 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbc30c7beaa041f0badbb147e304e138. Latest consensus state: current_term: 1 leader_uuid: "fbc30c7beaa041f0badbb147e304e138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc30c7beaa041f0badbb147e304e138" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33795 } } }
I20260925 09:15:01.068282 13607 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbc30c7beaa041f0badbb147e304e138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc30c7beaa041f0badbb147e304e138" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33795 } } }
I20260925 09:15:01.068852 13607 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [sys.catalog]: This master's current role is: LEADER
I20260925 09:15:01.068892 13608 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138 [sys.catalog]: This master's current role is: LEADER
I20260925 09:15:01.070906 13614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:15:01.081035 13614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:15:01.090874 13614 catalog_manager.cc:1347] Generated new cluster ID: 039d706e31c243bc93fdb8de0b476dbe
I20260925 09:15:01.091007 13614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:15:01.113963 13614 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:15:01.115361 13614 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:15:01.132068 13614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138: Generated new TSK 0
I20260925 09:15:01.133050 13614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:15:01.205780 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33795
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:15:01.548537 13627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:15:01.548858 13627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:15:01.549028 13627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:15:01.556720 13627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:15:01.556852 13627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:15:01.557039 13627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:15:01.567458 13627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:15:01.568815 13627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:15:01.570242 13627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:15:01.584482 13633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:01.584515 13632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:01.584753 13635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:15:01.586781 13627 server_base.cc:1034] running on GCE node
I20260925 09:15:01.587493 13627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:15:01.588647 13627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:15:01.589921 13627 hybrid_clock.cc:648] HybridClock initialized: now 1790327701589872 us; error 66 us; skew 500 ppm
I20260925 09:15:01.590312 13627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:15:01 dist-test-slave-l6qh krb5kdc[13520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327701, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:15:01.593845 13627 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:15:01.595517 13627 webserver.cc:460] Webserver started at http://127.3.19.65:43369/ using document root <none> and password file <none>
I20260925 09:15:01.596280 13627 fs_manager.cc:359] Metadata directory not provided
I20260925 09:15:01.596426 13627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:15:01.596740 13627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:15:01.600183 13627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "09548fbdb7f84f7e971baf37caf80d5a"
format_stamp: "Formatted at 2026-09-25 09:15:01 on dist-test-slave-l6qh"
server_key: "0132c04e2f37adb14a3db80eaeb3c841"
server_key_iv: "534c32ec64f7edca535d6a72f495a712"
server_key_version: "encryptionkey@0"
I20260925 09:15:01.601017 13627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "09548fbdb7f84f7e971baf37caf80d5a"
format_stamp: "Formatted at 2026-09-25 09:15:01 on dist-test-slave-l6qh"
server_key: "0132c04e2f37adb14a3db80eaeb3c841"
server_key_iv: "534c32ec64f7edca535d6a72f495a712"
server_key_version: "encryptionkey@0"
I20260925 09:15:01.606351 13627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 09:15:01.609979 13642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:01.611409 13627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:15:01.611603 13627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "09548fbdb7f84f7e971baf37caf80d5a"
format_stamp: "Formatted at 2026-09-25 09:15:01 on dist-test-slave-l6qh"
server_key: "0132c04e2f37adb14a3db80eaeb3c841"
server_key_iv: "534c32ec64f7edca535d6a72f495a712"
server_key_version: "encryptionkey@0"
I20260925 09:15:01.611838 13627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:15:01.650362 13627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:15:01.656032 13627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:15:01.656389 13627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:15:01.657712 13627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:15:01.659206 13627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:15:01.659307 13627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:01.659476 13627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:15:01.659543 13627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:01.696516 13627 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:40925
I20260925 09:15:01.696667 13754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:40925 every 8 connection(s)
I20260925 09:15:01.698041 13627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:15:01.703943 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 13627
I20260925 09:15:01.704200 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:15:01.704993 3149 external_mini_cluster.cc:1428] Setting key 2b18ea64051d879b601792248499e26b
Sep 25 09:15:01 dist-test-slave-l6qh krb5kdc[13520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327701, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:15:01.722083 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:01.700787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:51469 (local address 127.3.19.126:33795)
0925 09:15:01.701286 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:01.701292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:01.702373 (+ 1081us) server_negotiation.cc:407] Connection header received
0925 09:15:01.704718 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:01.704723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:01.704809 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:01.704915 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:01.706891 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:01.707764 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:01.710525 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:01.710966 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:01.711075 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:01.715815 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:01.715821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:01.715825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:01.715862 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:01.718891 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:01.719675 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:01.719681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:01.719684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:01.719813 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:01.720265 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:01.720275 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:01.720277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:01.720538 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:01.720715 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:01.721475 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 09:15:01.721713 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":235,"threads_started":1}
I20260925 09:15:01.723125 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:01.701466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33795 (local address 127.3.19.65:51469)
0925 09:15:01.702203 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:01.702247 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:01.704351 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:01.705253 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:01.705267 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:01.706015 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:15:01.706669 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:01.706720 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:01.707953 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:01.707960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:01.710345 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:01.710364 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:01.711217 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:01.711227 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:01.711564 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:01.712382 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:15:01.712425 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:15:01.715538 (+ 3113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:15:01.719051 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:15:01.719060 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:15:01.719090 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:15:01.719538 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:15:01.719929 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:15:01.719934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:15:01.719938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:15:01.720140 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:15:01.720678 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:15:01.720686 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:15:01.721043 (+ 357us) client_negotiation.cc:769] Sending connection context
0925 09:15:01.721378 (+ 335us) client_negotiation.cc:240] Negotiation successful
0925 09:15:01.721940 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":322,"threads_started":1}
I20260925 09:15:01.725050 13755 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33795
I20260925 09:15:01.725499 13755 heartbeater.cc:464] Registering TS with master...
I20260925 09:15:01.726300 13755 heartbeater.cc:511] Master 127.3.19.126:33795 requested a full tablet report, sending...
I20260925 09:15:01.728564 13565 ts_manager.cc:194] Registered new tserver with Master: 09548fbdb7f84f7e971baf37caf80d5a (127.3.19.65:40925)
I20260925 09:15:01.730170 13565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:51469
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:15:01.765934 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33795
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:15:02.108891 13761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:15:02.109426 13761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:15:02.109643 13761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:15:02.117462 13761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:15:02.117592 13761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:15:02.117792 13761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:15:02.127827 13761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:15:02.129202 13761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:15:02.130731 13761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:15:02.144589 13766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:02.144668 13769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:02.144668 13767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:15:02.145711 13761 server_base.cc:1034] running on GCE node
I20260925 09:15:02.146320 13761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:15:02.147266 13761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:15:02.148469 13761 hybrid_clock.cc:648] HybridClock initialized: now 1790327702148400 us; error 61 us; skew 500 ppm
I20260925 09:15:02.148810 13761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:15:02 dist-test-slave-l6qh krb5kdc[13520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327702, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:15:02.152159 13761 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:15:02.153645 13761 webserver.cc:460] Webserver started at http://127.3.19.66:37833/ using document root <none> and password file <none>
I20260925 09:15:02.154299 13761 fs_manager.cc:359] Metadata directory not provided
I20260925 09:15:02.154397 13761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:15:02.154666 13761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:15:02.157649 13761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "88152abf405548b0bbeb74fd16bf9790"
format_stamp: "Formatted at 2026-09-25 09:15:02 on dist-test-slave-l6qh"
server_key: "7646898371e2f7d947a726389d53aadb"
server_key_iv: "597141bcfead90ba8bab4b92638975c9"
server_key_version: "encryptionkey@0"
I20260925 09:15:02.158386 13761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "88152abf405548b0bbeb74fd16bf9790"
format_stamp: "Formatted at 2026-09-25 09:15:02 on dist-test-slave-l6qh"
server_key: "7646898371e2f7d947a726389d53aadb"
server_key_iv: "597141bcfead90ba8bab4b92638975c9"
server_key_version: "encryptionkey@0"
I20260925 09:15:02.163587 13761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 09:15:02.167140 13776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:02.168579 13761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:15:02.168784 13761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "88152abf405548b0bbeb74fd16bf9790"
format_stamp: "Formatted at 2026-09-25 09:15:02 on dist-test-slave-l6qh"
server_key: "7646898371e2f7d947a726389d53aadb"
server_key_iv: "597141bcfead90ba8bab4b92638975c9"
server_key_version: "encryptionkey@0"
I20260925 09:15:02.168967 13761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:15:02.181708 13761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:15:02.186125 13761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:15:02.186465 13761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:15:02.187417 13761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:15:02.188737 13761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:15:02.188807 13761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:02.188970 13761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:15:02.189020 13761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:02.222498 13761 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:39953
I20260925 09:15:02.222553 13889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:39953 every 8 connection(s)
I20260925 09:15:02.223850 13761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:15:02.231410 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 13761
I20260925 09:15:02.231639 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:15:02.232039 3149 external_mini_cluster.cc:1428] Setting key 5c6ca3a95bc8ddf36d8d0c12b77980f1
Sep 25 09:15:02 dist-test-slave-l6qh krb5kdc[13520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327702, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:15:02.245443 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.226503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:34705 (local address 127.3.19.126:33795)
0925 09:15:02.226981 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:02.226987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:02.227964 (+ 977us) server_negotiation.cc:407] Connection header received
0925 09:15:02.229806 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:02.229811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:02.229923 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:02.230041 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:02.231592 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.232402 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.234246 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.234851 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.234988 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.239050 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:02.239055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:02.239059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:02.239091 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:02.241792 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:02.242799 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:02.242805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:02.242810 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:02.242971 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:02.243397 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:02.243405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:02.243409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:02.243806 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:02.243977 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:02.244631 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 09:15:02.244893 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":246,"threads_started":1}
I20260925 09:15:02.246104 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.227126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33795 (local address 127.3.19.66:34705)
0925 09:15:02.227801 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:02.227839 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:02.229480 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:02.230297 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:02.230308 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:02.230829 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:15:02.231415 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.231431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.232543 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.232555 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.234096 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.234111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.235074 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.235082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.235315 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.236054 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:15:02.236087 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:15:02.238807 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:15:02.241935 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:15:02.241947 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:15:02.241967 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:15:02.242642 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:15:02.243093 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:15:02.243096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:15:02.243099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:15:02.243281 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:15:02.243926 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:15:02.243932 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:15:02.244264 (+ 332us) client_negotiation.cc:769] Sending connection context
0925 09:15:02.244490 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 09:15:02.244894 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":264,"threads_started":1}
I20260925 09:15:02.247644 13890 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33795
I20260925 09:15:02.247989 13890 heartbeater.cc:464] Registering TS with master...
I20260925 09:15:02.248888 13890 heartbeater.cc:511] Master 127.3.19.126:33795 requested a full tablet report, sending...
I20260925 09:15:02.250639 13565 ts_manager.cc:194] Registered new tserver with Master: 88152abf405548b0bbeb74fd16bf9790 (127.3.19.66:39953)
I20260925 09:15:02.251489 13565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:34705
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:15:02.289405 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33795
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:15:02.615338 13896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:15:02.615667 13896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:15:02.615875 13896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:15:02.622992 13896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:15:02.623111 13896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:15:02.623289 13896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:15:02.633774 13896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:15:02.635116 13896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:15:02.636394 13896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:15:02.649077 13902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:02.650131 13901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:02.650594 13904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:15:02.650997 13896 server_base.cc:1034] running on GCE node
I20260925 09:15:02.651974 13896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:15:02.653126 13896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:15:02.654332 13896 hybrid_clock.cc:648] HybridClock initialized: now 1790327702654267 us; error 52 us; skew 500 ppm
I20260925 09:15:02.654757 13896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:15:02 dist-test-slave-l6qh krb5kdc[13520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327702, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:15:02.658650 13896 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:15:02.661165 13896 webserver.cc:460] Webserver started at http://127.3.19.67:35971/ using document root <none> and password file <none>
I20260925 09:15:02.662201 13896 fs_manager.cc:359] Metadata directory not provided
I20260925 09:15:02.662356 13896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:15:02.662678 13896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:15:02.666972 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "bcd204955df944c3ad9a0648c7d56c88"
format_stamp: "Formatted at 2026-09-25 09:15:02 on dist-test-slave-l6qh"
server_key: "6b997a55a05e29839825a4e872a88bbc"
server_key_iv: "f4a7dc83cf76fa34173fb58af3321829"
server_key_version: "encryptionkey@0"
I20260925 09:15:02.667958 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "bcd204955df944c3ad9a0648c7d56c88"
format_stamp: "Formatted at 2026-09-25 09:15:02 on dist-test-slave-l6qh"
server_key: "6b997a55a05e29839825a4e872a88bbc"
server_key_iv: "f4a7dc83cf76fa34173fb58af3321829"
server_key_version: "encryptionkey@0"
I20260925 09:15:02.675335 13896 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260925 09:15:02.678584 13911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:02.679867 13896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:15:02.679987 13896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "bcd204955df944c3ad9a0648c7d56c88"
format_stamp: "Formatted at 2026-09-25 09:15:02 on dist-test-slave-l6qh"
server_key: "6b997a55a05e29839825a4e872a88bbc"
server_key_iv: "f4a7dc83cf76fa34173fb58af3321829"
server_key_version: "encryptionkey@0"
I20260925 09:15:02.680150 13896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:15:02.704226 13896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:15:02.707895 13896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:15:02.708134 13896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:15:02.708976 13896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:15:02.710253 13896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:15:02.710322 13896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:02.710400 13896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:15:02.710444 13896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:02.733843 13755 heartbeater.cc:503] Master 127.3.19.126:33795 was elected leader, sending a full tablet report...
I20260925 09:15:02.743577 13896 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:46499
I20260925 09:15:02.743642 14024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:46499 every 8 connection(s)
I20260925 09:15:02.744864 13896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:15:02.753328 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 13896
I20260925 09:15:02.753553 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:15:02.753969 3149 external_mini_cluster.cc:1428] Setting key 41b3507f8a7403a9b20f8ec25882a196
Sep 25 09:15:02 dist-test-slave-l6qh krb5kdc[13520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327702, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:15:02.764667 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.747474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:51953 (local address 127.3.19.126:33795)
0925 09:15:02.747881 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:02.747886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:02.748910 (+ 1024us) server_negotiation.cc:407] Connection header received
0925 09:15:02.750700 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:02.750705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:02.750786 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:02.750879 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:02.752322 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.753346 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.755325 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.755908 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.756032 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.759531 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:02.759541 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:02.759545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:02.759576 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:02.761916 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:02.762657 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:02.762661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:02.762663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:02.762778 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:02.763166 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:02.763169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:02.763171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:02.763414 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:02.763523 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:02.764229 (+ 706us) server_negotiation.cc:299] Negotiation successful
0925 09:15:02.764387 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":200,"threads_started":1}
I20260925 09:15:02.765720 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.748061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33795 (local address 127.3.19.67:51953)
0925 09:15:02.748757 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:02.748800 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:02.750442 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:02.751081 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:02.751089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:02.751571 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:15:02.752157 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.752179 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.753526 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.753530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.755175 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.755196 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.756164 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.756172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.756459 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.757179 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:15:02.757200 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:15:02.759335 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:15:02.762053 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:15:02.762060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:15:02.762073 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:15:02.762510 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:15:02.762880 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:15:02.762883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:15:02.762885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:15:02.763077 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:15:02.763580 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:15:02.763586 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:15:02.763900 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 09:15:02.764140 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 09:15:02.764539 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":252,"threads_started":1}
I20260925 09:15:02.767107 14025 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33795
I20260925 09:15:02.767427 14025 heartbeater.cc:464] Registering TS with master...
I20260925 09:15:02.768332 14025 heartbeater.cc:511] Master 127.3.19.126:33795 requested a full tablet report, sending...
I20260925 09:15:02.769958 13565 ts_manager.cc:194] Registered new tserver with Master: bcd204955df944c3ad9a0648c7d56c88 (127.3.19.67:46499)
I20260925 09:15:02.770788 13565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:51953
I20260925 09:15:02.782076 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:15:02.800069 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.787334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59952 (local address 127.3.19.126:33795)
0925 09:15:02.787566 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:02.787572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:02.788108 (+ 536us) server_negotiation.cc:407] Connection header received
0925 09:15:02.788310 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:02.788316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:02.788417 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:02.788530 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:02.790351 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.791129 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.792381 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.792821 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.792924 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.795206 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:02.795211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:02.795214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:02.795246 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:02.797860 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:02.798401 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:02.798405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:02.798407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:02.798516 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:02.798858 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:02.798861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:02.798863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:02.799170 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:02.799354 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:02.799665 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 09:15:02.799824 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 09:15:02.804077 13565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:15:02.811005 13565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:15:02.849606 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.829181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.3.19.67:46499)
0925 09:15:02.841431 (+ 12250us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:02.841444 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:02.841486 (+ 42us) server_negotiation.cc:407] Connection header received
0925 09:15:02.841581 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:02.841588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:02.841819 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:02.842018 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:15:02.842771 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.844246 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.846858 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.848343 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.848480 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.848718 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:02.849037 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 09:15:02.849222 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12019,"thread_start_us":11843,"threads_started":1}
I20260925 09:15:02.849701 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.828792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:46499 (local address 127.0.0.1:43698)
0925 09:15:02.840647 (+ 11855us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:02.840670 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:02.840802 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:02.842008 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:02.842012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:02.842039 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:15:02.842510 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.842527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.844372 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.844380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.846702 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.846713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.848562 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.848569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.848764 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.848839 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 09:15:02.848981 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 09:15:02.849184 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11698,"mutex_wait_us":70,"thread_start_us":11544,"threads_started":1}
I20260925 09:15:02.849756 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.823809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:40925 (local address 127.0.0.1:37826)
0925 09:15:02.838891 (+ 15082us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:02.838956 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:02.839186 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:02.842427 (+ 3241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:02.842437 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:02.842468 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:15:02.842875 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.842888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.845566 (+ 2678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.845571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.847475 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.847485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.848883 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.848888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.849062 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.849105 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 09:15:02.849195 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 09:15:02.849347 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14749,"mutex_wait_us":2,"thread_start_us":15601,"threads_started":1}
I20260925 09:15:02.850324 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.824087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.3.19.65:40925)
0925 09:15:02.841846 (+ 17759us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:02.841854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:02.841879 (+ 25us) server_negotiation.cc:407] Connection header received
0925 09:15:02.841971 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:02.841979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:02.842220 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:02.842429 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:15:02.843890 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.845393 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.847628 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.848710 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.848837 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.849074 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:02.849283 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 09:15:02.849457 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17590,"thread_start_us":17414,"threads_started":1}
I20260925 09:15:02.851861 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.824303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:39953 (local address 127.0.0.1:56464)
0925 09:15:02.840306 (+ 16003us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:02.840333 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:02.840733 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:02.841824 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:02.841831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:02.841863 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:15:02.842306 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.842330 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.843822 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.843829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.846802 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.846816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.851241 (+ 4425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.851246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.851377 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.851401 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 09:15:02.851485 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 09:15:02.851582 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15823,"mutex_wait_us":77,"thread_start_us":16175,"threads_started":1}
I20260925 09:15:02.852012 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.824805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.3.19.66:39953)
0925 09:15:02.835109 (+ 10304us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:02.835118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:02.840453 (+ 5335us) server_negotiation.cc:407] Connection header received
0925 09:15:02.840946 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:02.840955 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:02.841282 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:02.841476 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:15:02.842542 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.843687 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.846939 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.848342 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.848481 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.848715 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:02.851535 (+ 2820us) server_negotiation.cc:299] Negotiation successful
0925 09:15:02.851714 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10133,"thread_start_us":10007,"threads_started":1}
I20260925 09:15:02.853861 13959 tablet_service.cc:1467] Processing CreateTablet for tablet 36b65f7b2e384039ab03bb3c9cd1f8b3 (DEFAULT_TABLE table=test-table [id=cae450a8865848439034ec329169356c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:15:02.855159 13959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b65f7b2e384039ab03bb3c9cd1f8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:15:02.855391 13825 tablet_service.cc:1467] Processing CreateTablet for tablet 36b65f7b2e384039ab03bb3c9cd1f8b3 (DEFAULT_TABLE table=test-table [id=cae450a8865848439034ec329169356c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:15:02.856859 13825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b65f7b2e384039ab03bb3c9cd1f8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:15:02.857208 13690 tablet_service.cc:1467] Processing CreateTablet for tablet 36b65f7b2e384039ab03bb3c9cd1f8b3 (DEFAULT_TABLE table=test-table [id=cae450a8865848439034ec329169356c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:15:02.858695 13690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b65f7b2e384039ab03bb3c9cd1f8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:15:02.867138 14044 tablet_bootstrap.cc:492] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88: Bootstrap starting.
I20260925 09:15:02.870223 14044 tablet_bootstrap.cc:654] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88: Neither blocks nor log segments found. Creating new log.
I20260925 09:15:02.871392 14044 log.cc:824] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88: Log is configured to *not* fsync() on all Append() calls
I20260925 09:15:02.873472 14045 tablet_bootstrap.cc:492] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790: Bootstrap starting.
I20260925 09:15:02.875337 14044 tablet_bootstrap.cc:492] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88: No bootstrap required, opened a new log
I20260925 09:15:02.875609 14044 ts_tablet_manager.cc:1397] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 09:15:02.876971 14045 tablet_bootstrap.cc:654] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790: Neither blocks nor log segments found. Creating new log.
I20260925 09:15:02.877807 14047 tablet_bootstrap.cc:492] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a: Bootstrap starting.
I20260925 09:15:02.878180 14045 log.cc:824] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790: Log is configured to *not* fsync() on all Append() calls
I20260925 09:15:02.881067 14047 tablet_bootstrap.cc:654] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a: Neither blocks nor log segments found. Creating new log.
I20260925 09:15:02.881145 14045 tablet_bootstrap.cc:492] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790: No bootstrap required, opened a new log
I20260925 09:15:02.881399 14045 ts_tablet_manager.cc:1397] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 09:15:02.882162 14047 log.cc:824] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a: Log is configured to *not* fsync() on all Append() calls
I20260925 09:15:02.885368 14047 tablet_bootstrap.cc:492] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a: No bootstrap required, opened a new log
I20260925 09:15:02.885646 14047 ts_tablet_manager.cc:1397] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 09:15:02.885892 14044 raft_consensus.cc:348] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09548fbdb7f84f7e971baf37caf80d5a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 40925 } } peers { permanent_uuid: "88152abf405548b0bbeb74fd16bf9790" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 39953 } } peers { permanent_uuid: "bcd204955df944c3ad9a0648c7d56c88" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 46499 } }
I20260925 09:15:02.886297 14044 raft_consensus.cc:374] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:15:02.886391 14044 raft_consensus.cc:729] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcd204955df944c3ad9a0648c7d56c88, State: Initialized, Role: FOLLOWER
I20260925 09:15:02.886996 14044 consensus_queue.cc:260] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09548fbdb7f84f7e971baf37caf80d5a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 40925 } } peers { permanent_uuid: "88152abf405548b0bbeb74fd16bf9790" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 39953 } } peers { permanent_uuid: "bcd204955df944c3ad9a0648c7d56c88" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 46499 } }
I20260925 09:15:02.888094 14025 heartbeater.cc:503] Master 127.3.19.126:33795 was elected leader, sending a full tablet report...
I20260925 09:15:02.888652 14044 ts_tablet_manager.cc:1428] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 09:15:02.891388 14045 raft_consensus.cc:348] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09548fbdb7f84f7e971baf37caf80d5a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 40925 } } peers { permanent_uuid: "88152abf405548b0bbeb74fd16bf9790" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 39953 } } peers { permanent_uuid: "bcd204955df944c3ad9a0648c7d56c88" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 46499 } }
I20260925 09:15:02.891821 14045 raft_consensus.cc:374] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:15:02.891906 14045 raft_consensus.cc:729] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88152abf405548b0bbeb74fd16bf9790, State: Initialized, Role: FOLLOWER
I20260925 09:15:02.892525 14045 consensus_queue.cc:260] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09548fbdb7f84f7e971baf37caf80d5a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 40925 } } peers { permanent_uuid: "88152abf405548b0bbeb74fd16bf9790" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 39953 } } peers { permanent_uuid: "bcd204955df944c3ad9a0648c7d56c88" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 46499 } }
I20260925 09:15:02.893762 13890 heartbeater.cc:503] Master 127.3.19.126:33795 was elected leader, sending a full tablet report...
I20260925 09:15:02.894399 14045 ts_tablet_manager.cc:1428] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260925 09:15:02.895285 14047 raft_consensus.cc:348] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09548fbdb7f84f7e971baf37caf80d5a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 40925 } } peers { permanent_uuid: "88152abf405548b0bbeb74fd16bf9790" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 39953 } } peers { permanent_uuid: "bcd204955df944c3ad9a0648c7d56c88" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 46499 } }
I20260925 09:15:02.895599 14047 raft_consensus.cc:374] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:15:02.895732 14047 raft_consensus.cc:729] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09548fbdb7f84f7e971baf37caf80d5a, State: Initialized, Role: FOLLOWER
I20260925 09:15:02.896395 14047 consensus_queue.cc:260] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09548fbdb7f84f7e971baf37caf80d5a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 40925 } } peers { permanent_uuid: "88152abf405548b0bbeb74fd16bf9790" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 39953 } } peers { permanent_uuid: "bcd204955df944c3ad9a0648c7d56c88" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 46499 } }
I20260925 09:15:02.898214 14047 ts_tablet_manager.cc:1428] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260925 09:15:02.923877 14052 raft_consensus.cc:482] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:15:02.924137 14052 raft_consensus.cc:504] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09548fbdb7f84f7e971baf37caf80d5a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 40925 } } peers { permanent_uuid: "88152abf405548b0bbeb74fd16bf9790" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 39953 } } peers { permanent_uuid: "bcd204955df944c3ad9a0648c7d56c88" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 46499 } }
I20260925 09:15:02.925837 14052 leader_election.cc:288] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88152abf405548b0bbeb74fd16bf9790 (127.3.19.66:39953), bcd204955df944c3ad9a0648c7d56c88 (127.3.19.67:46499)
I20260925 09:15:02.932528 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.926244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:54485 (local address 127.3.19.66:39953)
0925 09:15:02.926469 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:02.926477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:02.927245 (+ 768us) server_negotiation.cc:407] Connection header received
0925 09:15:02.927446 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:02.927451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:02.927515 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:02.927606 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:15:02.928200 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.928894 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.930756 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.931505 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.931666 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.931772 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:02.932160 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 09:15:02.932287 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 09:15:02.932613 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.926246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:36245 (local address 127.3.19.67:46499)
0925 09:15:02.926443 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:02.926452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:02.927287 (+ 835us) server_negotiation.cc:407] Connection header received
0925 09:15:02.927483 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:02.927487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:02.927547 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:02.927645 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:15:02.928269 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.928989 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.930783 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.931675 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.931819 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.931901 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:02.932284 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 09:15:02.932400 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 09:15:02.932699 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.926237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:46499 (local address 127.3.19.65:36245)
0925 09:15:02.927179 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:02.927194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:02.927340 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:02.927659 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:02.927663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:02.927684 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:15:02.928086 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.928095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.929133 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.929138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.930638 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.930649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.931920 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.931925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.932147 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.932175 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 09:15:02.932249 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 09:15:02.932341 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":51,"thread_start_us":441,"threads_started":1}
I20260925 09:15:02.933271 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.926159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:39953 (local address 127.3.19.65:54485)
0925 09:15:02.927114 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:02.927136 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:02.927303 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:02.927631 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:02.927635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:02.927657 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:15:02.928056 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.928064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.929028 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.929033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.930611 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:02.930624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.931711 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.931718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:02.931911 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.932031 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 09:15:02.932132 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 09:15:02.932262 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":38,"thread_start_us":686,"threads_started":1}
I20260925 09:15:02.933756 13979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b65f7b2e384039ab03bb3c9cd1f8b3" candidate_uuid: "09548fbdb7f84f7e971baf37caf80d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcd204955df944c3ad9a0648c7d56c88" is_pre_election: true
I20260925 09:15:02.934237 13845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b65f7b2e384039ab03bb3c9cd1f8b3" candidate_uuid: "09548fbdb7f84f7e971baf37caf80d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88152abf405548b0bbeb74fd16bf9790" is_pre_election: true
I20260925 09:15:02.934341 13979 raft_consensus.cc:2454] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09548fbdb7f84f7e971baf37caf80d5a in term 0.
I20260925 09:15:02.934685 13845 raft_consensus.cc:2454] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09548fbdb7f84f7e971baf37caf80d5a in term 0.
I20260925 09:15:02.935019 13646 leader_election.cc:302] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09548fbdb7f84f7e971baf37caf80d5a, bcd204955df944c3ad9a0648c7d56c88; no voters:
I20260925 09:15:02.935386 14052 raft_consensus.cc:2781] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:15:02.935524 14052 raft_consensus.cc:482] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:15:02.935636 14052 raft_consensus.cc:3037] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:15:02.937937 14052 raft_consensus.cc:504] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09548fbdb7f84f7e971baf37caf80d5a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 40925 } } peers { permanent_uuid: "88152abf405548b0bbeb74fd16bf9790" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 39953 } } peers { permanent_uuid: "bcd204955df944c3ad9a0648c7d56c88" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 46499 } }
I20260925 09:15:02.938452 14052 leader_election.cc:288] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [CANDIDATE]: Term 1 election: Requested vote from peers 88152abf405548b0bbeb74fd16bf9790 (127.3.19.66:39953), bcd204955df944c3ad9a0648c7d56c88 (127.3.19.67:46499)
I20260925 09:15:02.938819 13845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b65f7b2e384039ab03bb3c9cd1f8b3" candidate_uuid: "09548fbdb7f84f7e971baf37caf80d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88152abf405548b0bbeb74fd16bf9790"
I20260925 09:15:02.938894 13979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b65f7b2e384039ab03bb3c9cd1f8b3" candidate_uuid: "09548fbdb7f84f7e971baf37caf80d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcd204955df944c3ad9a0648c7d56c88"
I20260925 09:15:02.939005 13845 raft_consensus.cc:3037] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:15:02.939096 13979 raft_consensus.cc:3037] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:15:02.941295 13845 raft_consensus.cc:2454] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09548fbdb7f84f7e971baf37caf80d5a in term 1.
I20260925 09:15:02.941483 13979 raft_consensus.cc:2454] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09548fbdb7f84f7e971baf37caf80d5a in term 1.
I20260925 09:15:02.941669 13644 leader_election.cc:302] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09548fbdb7f84f7e971baf37caf80d5a, 88152abf405548b0bbeb74fd16bf9790; no voters:
I20260925 09:15:02.941955 14052 raft_consensus.cc:2781] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:15:02.942374 14052 raft_consensus.cc:686] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [term 1 LEADER]: Becoming Leader. State: Replica: 09548fbdb7f84f7e971baf37caf80d5a, State: Running, Role: LEADER
I20260925 09:15:02.942857 14052 consensus_queue.cc:237] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09548fbdb7f84f7e971baf37caf80d5a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 40925 } } peers { permanent_uuid: "88152abf405548b0bbeb74fd16bf9790" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 39953 } } peers { permanent_uuid: "bcd204955df944c3ad9a0648c7d56c88" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 46499 } }
I20260925 09:15:02.947152 13564 catalog_manager.cc:5495] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 09548fbdb7f84f7e971baf37caf80d5a (127.3.19.65). New cstate: current_term: 1 leader_uuid: "09548fbdb7f84f7e971baf37caf80d5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09548fbdb7f84f7e971baf37caf80d5a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88152abf405548b0bbeb74fd16bf9790" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 39953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcd204955df944c3ad9a0648c7d56c88" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 46499 } health_report { overall_health: UNKNOWN } } }
W20260925 09:15:02.950577 13756 tablet.cc:2369] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:15:02.969521 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:02.964386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.3.19.65:40925)
0925 09:15:02.964614 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:02.964619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:02.964638 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:15:02.964751 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:02.964755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:02.964832 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:02.964963 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:15:02.965336 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.966103 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.967419 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:02.967856 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:02.967960 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:02.968429 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:15:02.968943 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:15:02.969060 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:02.969128 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 09:15:02.969188 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 09:15:02.975816 13891 tablet.cc:2369] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:15:02.977764 13979 raft_consensus.cc:1261] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88 [term 1 FOLLOWER]: Refusing update from remote peer 09548fbdb7f84f7e971baf37caf80d5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:15:02.977854 13845 raft_consensus.cc:1261] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790 [term 1 FOLLOWER]: Refusing update from remote peer 09548fbdb7f84f7e971baf37caf80d5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:15:02.978724 14052 consensus_queue.cc:1035] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd204955df944c3ad9a0648c7d56c88" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 46499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:15:02.979064 14055 consensus_queue.cc:1035] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "88152abf405548b0bbeb74fd16bf9790" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 39953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 09:15:02.996419 14026 tablet.cc:2369] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:15:02.997387 14060 mvcc.cc:205] Tried to move back new op lower bound from 7333182271386423296 to 7333182271257391104. Current Snapshot: MvccSnapshot[applied={T|T < 7333182271386423296}]
I20260925 09:15:02.999387 14061 mvcc.cc:205] Tried to move back new op lower bound from 7333182271386423296 to 7333182271257391104. Current Snapshot: MvccSnapshot[applied={T|T < 7333182271386423296}]
I20260925 09:15:03.009207 13564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:15:03.009557 13564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:15:03.013186 13564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbc30c7beaa041f0badbb147e304e138: Sending DeleteTablet for 3 replicas of tablet 36b65f7b2e384039ab03bb3c9cd1f8b3
I20260925 09:15:03.014355 13690 tablet_service.cc:1514] Processing DeleteTablet for tablet 36b65f7b2e384039ab03bb3c9cd1f8b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:15:03 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:37826
I20260925 09:15:03.014348 13825 tablet_service.cc:1514] Processing DeleteTablet for tablet 36b65f7b2e384039ab03bb3c9cd1f8b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:15:03 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:56464
I20260925 09:15:03.014474 13959 tablet_service.cc:1514] Processing DeleteTablet for tablet 36b65f7b2e384039ab03bb3c9cd1f8b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:15:03 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:43698
I20260925 09:15:03.017428 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 13627
I20260925 09:15:03.027136 14070 tablet_replica.cc:330] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790: stopping tablet replica
I20260925 09:15:03.027535 14070 raft_consensus.cc:2229] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:15:03.027987 14070 raft_consensus.cc:2258] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:15:03.032413 14070 ts_tablet_manager.cc:1905] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:15:03.038054 14069 tablet_replica.cc:330] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a: stopping tablet replica
I20260925 09:15:03.038542 14069 raft_consensus.cc:2229] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:15:03.038750 14071 tablet_replica.cc:330] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88: stopping tablet replica
I20260925 09:15:03.039197 14071 raft_consensus.cc:2229] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:15:03.039230 14069 raft_consensus.cc:2258] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:15:03.040378 14071 raft_consensus.cc:2258] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:15:03.041860 14070 ts_tablet_manager.cc:1918] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:15:03.042034 14070 log.cc:1196] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/36b65f7b2e384039ab03bb3c9cd1f8b3
I20260925 09:15:03.042591 14070 ts_tablet_manager.cc:1939] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 88152abf405548b0bbeb74fd16bf9790: Deleting consensus metadata
I20260925 09:15:03.042599 14069 ts_tablet_manager.cc:1905] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:15:03.042989 14071 ts_tablet_manager.cc:1905] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:15:03.044346 13550 catalog_manager.cc:4841] TS 88152abf405548b0bbeb74fd16bf9790 (127.3.19.66:39953): tablet 36b65f7b2e384039ab03bb3c9cd1f8b3 (table test-table [id=cae450a8865848439034ec329169356c]) successfully deleted
Sep 25 09:15:03 dist-test-slave-l6qh krb5kdc[13520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:15:03.050680 14069 ts_tablet_manager.cc:1918] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:15:03.050854 14069 log.cc:1196] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/36b65f7b2e384039ab03bb3c9cd1f8b3
I20260925 09:15:03.051491 14069 ts_tablet_manager.cc:1939] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P 09548fbdb7f84f7e971baf37caf80d5a: Deleting consensus metadata
I20260925 09:15:03.053448 13549 catalog_manager.cc:4841] TS 09548fbdb7f84f7e971baf37caf80d5a (127.3.19.65:40925): tablet 36b65f7b2e384039ab03bb3c9cd1f8b3 (table test-table [id=cae450a8865848439034ec329169356c]) successfully deleted
I20260925 09:15:03.056321 14071 ts_tablet_manager.cc:1918] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:15:03.056479 14071 log.cc:1196] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/36b65f7b2e384039ab03bb3c9cd1f8b3
I20260925 09:15:03.056939 14071 ts_tablet_manager.cc:1939] T 36b65f7b2e384039ab03bb3c9cd1f8b3 P bcd204955df944c3ad9a0648c7d56c88: Deleting consensus metadata
I20260925 09:15:03.057610 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:03.020021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.3.19.65:40925)
0925 09:15:03.020494 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:03.020502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:03.022796 (+ 2294us) server_negotiation.cc:407] Connection header received
0925 09:15:03.022957 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:03.022962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:03.023048 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:03.023153 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:03.033661 (+ 10508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:03.035038 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:03.038976 (+ 3938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:03.039849 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:03.039960 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:03.049555 (+ 9595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:03.049568 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:03.049586 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:03.049642 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:03.053759 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:03.054683 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:03.054693 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:03.054697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:03.054914 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:03.055441 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:03.055447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:03.055451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:03.055916 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:03.056241 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:03.056876 (+ 635us) server_negotiation.cc:299] Negotiation successful
0925 09:15:03.057139 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260925 09:15:03.058548 13552 catalog_manager.cc:4841] TS bcd204955df944c3ad9a0648c7d56c88 (127.3.19.67:46499): tablet 36b65f7b2e384039ab03bb3c9cd1f8b3 (table test-table [id=cae450a8865848439034ec329169356c]) successfully deleted
I20260925 09:15:03.059049 13750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:15:03.159363 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 13627
I20260925 09:15:03.178375 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 13761
Sep 25 09:15:03 dist-test-slave-l6qh krb5kdc[13520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:15:03.193090 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:03.179144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.3.19.66:39953)
0925 09:15:03.179348 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:03.179354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:03.179374 (+ 20us) server_negotiation.cc:407] Connection header received
0925 09:15:03.179455 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:03.179460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:03.179529 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:03.179653 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:03.180852 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:03.181692 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:03.182972 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:03.183423 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:03.183601 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:03.187756 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:03.187769 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:03.187785 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:03.187820 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:03.190408 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:03.191079 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:03.191087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:03.191089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:03.191211 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:03.191573 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:03.191578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:03.191583 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:03.191956 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:03.192261 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:03.192540 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 09:15:03.192750 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 09:15:03.193820 13885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:15:03.280651 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 13761
I20260925 09:15:03.298293 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 13896
Sep 25 09:15:03 dist-test-slave-l6qh krb5kdc[13520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:15:03.315394 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:03.298997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.3.19.67:46499)
0925 09:15:03.299181 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:03.299185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:03.299203 (+ 18us) server_negotiation.cc:407] Connection header received
0925 09:15:03.299303 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:03.299307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:03.299367 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:03.299446 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:03.300705 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:03.301470 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:03.302742 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:03.303150 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:03.303242 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:03.310390 (+ 7148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:03.310405 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:03.310422 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:03.310458 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:03.313047 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:03.313705 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:03.313712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:03.313715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:03.313859 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:03.314187 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:03.314190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:03.314192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:03.314521 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:03.314730 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:03.314977 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 09:15:03.315145 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 09:15:03.316095 14020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:15:03.403832 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 13896
I20260925 09:15:03.420540 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 13533
I20260925 09:15:03.421708 13596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:15:03.526645 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 13533
2026-09-25T09:15:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4339 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:15:03 dist-test-slave-l6qh krb5kdc[14079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:15:03 dist-test-slave-l6qh krb5kdc[14079](info): set up 2 sockets
Sep 25 09:15:03 dist-test-slave-l6qh krb5kdc[14079](info): commencing operation
krb5kdc: starting...
W20260925 09:15:04.627106 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:15:04 dist-test-slave-l6qh krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:15:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:15:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/rangeradmin_127.3.19.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/rangeradmin_127.3.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/rangerlookup_127.3.19.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/rangerlookup_127.3.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/HTTP_127.3.19.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/HTTP_127.3.19.84@KRBTEST.COM.keytab.
I20260925 09:15:04.947645 3149 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCZfsc6/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/postgres -l logfile start
2026-09-25 09:15:07.433 UTC [14109] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 09:15:07.433 UTC [14109] LOG: listening on IPv4 address "127.3.19.84", port 57909
2026-09-25 09:15:07.436 UTC [14109] LOG: listening on Unix socket "/tmp/.s.PGSQL.57909"
2026-09-25 09:15:07.458 UTC [14112] LOG: database system was shut down at 2026-09-25 09:15:06 UTC
2026-09-25 09:15:07.461 UTC [14109] LOG: database system is ready to accept connections
I20260925 09:15:08.430850 3149 mini_postgres.cc:96] Postgres bound to 57909
2026-09-25 09:15:08.450 UTC [14120] FATAL: database "slave" does not exist
127.3.19.84:57909 - accepting connections
I20260925 09:15:08.451493 3149 mini_ranger.cc:161] Starting Ranger...
I20260925 09:15:08.478804 3149 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 09:15:08.948906 3149 mini_ranger.cc:87] Created ranger Postgres database
I20260925 09:15:08.949055 3149 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-admin
2026-09-25 09:15:09,300 [I] DB FLAVOR :POSTGRES
2026-09-25 09:15:09,300 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 09:15:09,300 [I] Checking connection..
2026-09-25 09:15:09,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 09:15:09,565 [I] Checking connection passed.
2026-09-25 09:15:09,565 [I] --------- Verifying version history table ---------
2026-09-25 09:15:09,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 09:15:09,813 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 09:15:09,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 09:15:10,057 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 09:15:10,057 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 09:15:10,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 09:15:10.310 UTC [14185] WARNING: there is no transaction in progress
2026-09-25 09:15:10,316 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 09:15:10,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 09:15:10,584 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 09:15:10,585 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 09:15:10,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 09:15:10,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 09:15:11,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-l6qh.c.gcp-upstream.internal','N') ;"
2026-09-25 09:15:11,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 09:15:11,567 [I] Table x_portal_user does not exist in database ranger
2026-09-25 09:15:11,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 09:15:11,811 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 09:15:11,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 09:15:12,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 09:15:12,303 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 09:15:12,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 09:15:13.244 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.260 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.276 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.290 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.309 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.325 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.377 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.383 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.396 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.406 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.415 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.429 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.438 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.444 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.450 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:13.456 UTC [14294] WARNING: there is no transaction in progress
2026-09-25 09:15:14,061 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 09:15:14,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 09:15:14,291 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 09:15:14,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 09:15:14,551 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 09:15:14,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 09:15:14,790 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 09:15:14,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 09:15:15,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 09:15:15,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-l6qh.c.gcp-upstream.internal';"
2026-09-25 09:15:15,487 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 09:15:15,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 09:15:15,722 [I] DB_PATCHES have already been applied
I20260925 09:15:15.728534 3149 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCZfsc6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 09:15:15.728619 3149 mini_ranger.cc:193] Using host: 127.3.19.84
I20260925 09:15:15.744722 3149 mini_ranger.cc:227] Ranger admin URL: http://127.3.19.84:38153
Sep 25, 2026 9:15:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 9:15:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 9:15:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 9:15:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 9:15:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 9:15:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 9:15:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 09:15:16.885979 3149 mini_ranger.cc:160] Time spent starting Ranger: real 8.435s user 0.002s sys 0.042s
Sep 25, 2026 9:15:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 9:15:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 9:15:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 9:15:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 9:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 9:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 9:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 9:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 9:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 9:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 9:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 9:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 9:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 9:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 9:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 9:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 9:15:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 9:15:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 09:15:38.644518 3149 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/rangerkms_127.3.19.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/rangerkms_127.3.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.19.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.19.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.19.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/HTTP_127.3.19.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.19.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/HTTP_127.3.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 09:15:38 dist-test-slave-l6qh krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327738, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 09:15:38.805742 3149 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 09:15:38.827461 3149 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 09:15:39.685727 3149 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 09:15:39.685900 3149 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-kms
I20260925 09:15:39.686025 3149 mini_ranger_kms.cc:227] Using postgres at 127.3.19.84:57909
2026-09-25 09:15:40,125 [I] DB FLAVOR :POSTGRES
2026-09-25 09:15:40,125 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 09:15:40,125 [I] Checking connection
2026-09-25 09:15:40,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 09:15:40,354 [I] connection success
2026-09-25 09:15:40,354 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 09:15:40,354 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 09:15:40,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 09:15:40,610 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 09:15:40,611 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 09:15:40,611 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 09:15:40,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:57909/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 09:15:40,858 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 09:15:41.320290 3149 mini_ranger_kms.cc:310] Created kms service
I20260925 09:15:41.501578 3149 mini_ranger_kms.cc:326] Created kudu user
I20260925 09:15:41.544546 3149 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 09:15:41 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327738, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:15:41.920621 3149 mini_ranger_kms.cc:384] Added ranger policy
I20260925 09:15:41.920943 3149 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 09:15:41.921017 3149 mini_ranger_kms.cc:240] Using host: 127.3.19.84
I20260925 09:15:41.946020 3149 mini_ranger_kms.cc:276] Ranger KMS PID: 14475
I20260925 09:15:41.946282 3149 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.19.84:59947
Sep 25, 2026 9:15:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 9:15:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 9:15:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 9:15:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 9:15:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 9:15:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 09:15:44.267445 3149 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.462s user 0.129s sys 0.330s
I20260925 09:15:44.267699 3149 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 09:15:44.267891 3149 mini_ranger_kms.cc:401] 127.3.19.84:59947/kms/v1/keys
Sep 25 09:15:44 dist-test-slave-l6qh krb5kdc[14079](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790327744, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.19.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 09:15:44 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327744, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.19.84@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
Sep 25 09:15:45 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327744, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.19.84@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
Sep 25 09:15:47 dist-test-slave-l6qh krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:15:47.444901 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33297
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:33297
--ranger_config_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.19.84:59947/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:15:47.817399 14526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:15:47.817700 14526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:15:47.817854 14526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:15:47.825584 14526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:15:47.825691 14526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:15:47.825764 14526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:15:47.825858 14526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:15:47.825920 14526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:15:47.835718 14526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.19.84:59947/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:33297
--ranger_config_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:15:47.837123 14526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:15:47.838413 14526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:15:47.850998 14534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:47.850957 14532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:47.851428 14531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:15:47.852531 14526 server_base.cc:1034] running on GCE node
I20260925 09:15:47.853350 14526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:15:47.854811 14526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:15:47.856019 14526 hybrid_clock.cc:648] HybridClock initialized: now 1790327747855976 us; error 44 us; skew 500 ppm
I20260925 09:15:47.856359 14526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:15:47 dist-test-slave-l6qh krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327747, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:15:47.862556 14526 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:15:47.864316 14526 webserver.cc:460] Webserver started at http://127.3.19.126:46533/ using document root <none> and password file <none>
I20260925 09:15:47.864986 14526 fs_manager.cc:359] Metadata directory not provided
I20260925 09:15:47.865161 14526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:15:47.865516 14526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 09:15:47 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327747, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:15:47.920526 14526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "b80f833f097e469aaa581a1e034079b8"
format_stamp: "Formatted at 2026-09-25 09:15:47 on dist-test-slave-l6qh"
server_key: "f313b15ba0bf3cc57d8b9c7297b8083f"
server_key_iv: "6be9e1395c30c6e08bf8466ddae4fd6b"
server_key_version: "kuduclusterkey@0"
I20260925 09:15:47.921445 14526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "b80f833f097e469aaa581a1e034079b8"
format_stamp: "Formatted at 2026-09-25 09:15:47 on dist-test-slave-l6qh"
server_key: "f313b15ba0bf3cc57d8b9c7297b8083f"
server_key_iv: "6be9e1395c30c6e08bf8466ddae4fd6b"
server_key_version: "kuduclusterkey@0"
I20260925 09:15:47.926770 14526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 09:15:47.946439 14541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:47.948238 14526 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 09:15:47.948424 14526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "b80f833f097e469aaa581a1e034079b8"
format_stamp: "Formatted at 2026-09-25 09:15:47 on dist-test-slave-l6qh"
server_key: "f313b15ba0bf3cc57d8b9c7297b8083f"
server_key_iv: "6be9e1395c30c6e08bf8466ddae4fd6b"
server_key_version: "kuduclusterkey@0"
I20260925 09:15:47.948698 14526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:15:47.972589 14526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:15:47.976181 14526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:15:47.976440 14526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:15:47.994937 14526 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:33297
I20260925 09:15:47.995029 14593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:33297 every 8 connection(s)
I20260925 09:15:47.996261 14526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:15:47.999815 14594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:15:48.001327 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 14526
I20260925 09:15:48.001513 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:15:48.002171 3149 external_mini_cluster.cc:1428] Setting key d9399b718a9516ef57a1b658bd922215
I20260925 09:15:48.012173 14594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8: Bootstrap starting.
I20260925 09:15:48.016114 14594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8: Neither blocks nor log segments found. Creating new log.
Sep 25 09:15:48 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:15:48.017194 14594 log.cc:824] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8: Log is configured to *not* fsync() on all Append() calls
I20260925 09:15:48.020563 14594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8: No bootstrap required, opened a new log
I20260925 09:15:48.025556 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:48.004253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.3.19.126:33297)
0925 09:15:48.005457 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:48.005470 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:48.005512 (+ 42us) server_negotiation.cc:407] Connection header received
0925 09:15:48.007151 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:48.007172 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:48.007532 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:48.007844 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:48.009415 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:48.010756 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:48.012525 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:48.013211 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:48.013421 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:48.018003 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:48.018011 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:48.018028 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:48.018076 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:48.021688 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:48.022372 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:48.022386 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:48.022391 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:48.022540 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:48.022918 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:48.022923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:48.022937 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:48.023402 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:48.023706 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:48.024105 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 09:15:48.024479 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":748,"threads_started":1}
I20260925 09:15:48.028064 14594 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80f833f097e469aaa581a1e034079b8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33297 } }
I20260925 09:15:48.028424 14594 raft_consensus.cc:374] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:15:48.028522 14594 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80f833f097e469aaa581a1e034079b8, State: Initialized, Role: FOLLOWER
I20260925 09:15:48.029014 14594 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80f833f097e469aaa581a1e034079b8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33297 } }
I20260925 09:15:48.029265 14594 raft_consensus.cc:388] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:15:48.029428 14594 raft_consensus.cc:482] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:15:48.029596 14594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:15:48.031791 14594 raft_consensus.cc:504] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80f833f097e469aaa581a1e034079b8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33297 } }
I20260925 09:15:48.032721 14594 leader_election.cc:302] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b80f833f097e469aaa581a1e034079b8; no voters:
I20260925 09:15:48.033375 14594 leader_election.cc:288] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:15:48.033491 14599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:15:48.034247 14599 raft_consensus.cc:686] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [term 1 LEADER]: Becoming Leader. State: Replica: b80f833f097e469aaa581a1e034079b8, State: Running, Role: LEADER
I20260925 09:15:48.034795 14599 consensus_queue.cc:237] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80f833f097e469aaa581a1e034079b8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33297 } }
I20260925 09:15:48.035616 14594 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:15:48.037755 14600 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b80f833f097e469aaa581a1e034079b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80f833f097e469aaa581a1e034079b8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33297 } } }
I20260925 09:15:48.038143 14600 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [sys.catalog]: This master's current role is: LEADER
I20260925 09:15:48.038909 14601 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b80f833f097e469aaa581a1e034079b8. Latest consensus state: current_term: 1 leader_uuid: "b80f833f097e469aaa581a1e034079b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80f833f097e469aaa581a1e034079b8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33297 } } }
I20260925 09:15:48.039233 14601 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8 [sys.catalog]: This master's current role is: LEADER
I20260925 09:15:48.046327 14607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:15:48.051687 14607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:15:48.052268 14594 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 09:15:48.081979 14607 catalog_manager.cc:1347] Generated new cluster ID: 5b836994b81f4fc0be96131c8a3b42e6
I20260925 09:15:48.082433 14607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:15:48.098948 14607 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:15:48.100934 14607 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:15:48.116755 14607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8: Generated new TSK 0
I20260925 09:15:48.118014 14607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:15:48.924916 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.19.84:59947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33297
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 09:15:49 dist-test-slave-l6qh krb5kdc[14079](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790327749, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 09:15:49.325320 14639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:15:49.325604 14639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:15:49.325806 14639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:15:49.333032 14639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:15:49.333170 14639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:15:49.333345 14639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:15:49.347314 14639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.19.84:59947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:15:49.348945 14639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:15:49.350709 14639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:15:49.368510 14650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:49.369097 14649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:15:49.377591 14639 server_base.cc:1034] running on GCE node
W20260925 09:15:49.377930 14652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:15:49.378756 14639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:15:49.380159 14639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:15:49.381430 14639 hybrid_clock.cc:648] HybridClock initialized: now 1790327749381356 us; error 63 us; skew 500 ppm
I20260925 09:15:49.382298 14639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:15:49 dist-test-slave-l6qh krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327749, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:15:49.386591 14639 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:15:49.392402 14639 webserver.cc:460] Webserver started at http://127.3.19.65:41631/ using document root <none> and password file <none>
I20260925 09:15:49.393445 14639 fs_manager.cc:359] Metadata directory not provided
I20260925 09:15:49.393615 14639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:15:49.394018 14639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 09:15:49 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327749, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:15:49.420565 14639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "a5719cefab6b4f18bb8d162ac22774ed"
format_stamp: "Formatted at 2026-09-25 09:15:49 on dist-test-slave-l6qh"
server_key: "87db2588489834e586fa66ead038807c"
server_key_iv: "ddb001e0aa2e5141948f7886639ab175"
server_key_version: "kuduclusterkey@0"
I20260925 09:15:49.421855 14639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "a5719cefab6b4f18bb8d162ac22774ed"
format_stamp: "Formatted at 2026-09-25 09:15:49 on dist-test-slave-l6qh"
server_key: "87db2588489834e586fa66ead038807c"
server_key_iv: "ddb001e0aa2e5141948f7886639ab175"
server_key_version: "kuduclusterkey@0"
I20260925 09:15:49.435827 14639 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.002s sys 0.007s
I20260925 09:15:49.468683 14659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:49.471293 14639 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.004s sys 0.006s
I20260925 09:15:49.471568 14639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "a5719cefab6b4f18bb8d162ac22774ed"
format_stamp: "Formatted at 2026-09-25 09:15:49 on dist-test-slave-l6qh"
server_key: "87db2588489834e586fa66ead038807c"
server_key_iv: "ddb001e0aa2e5141948f7886639ab175"
server_key_version: "kuduclusterkey@0"
I20260925 09:15:49.471925 14639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:15:49.521739 14639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:15:49.528314 14639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:15:49.529120 14639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:15:49.530670 14639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:15:49.533157 14639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:15:49.539299 14639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260925 09:15:49.539564 14639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:15:49.539673 14639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:49.624552 14639 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:38147
I20260925 09:15:49.624591 14771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:38147 every 8 connection(s)
I20260925 09:15:49.626296 14639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:15:49.629323 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 14639
I20260925 09:15:49.629582 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:15:49.630566 3149 external_mini_cluster.cc:1428] Setting key adf10fa262b21ecfacd04cc0fa12aa56
Sep 25 09:15:49 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327749, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
I20260925 09:15:49.678602 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:49.631826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33297 (local address 127.3.19.65:33595)
0925 09:15:49.643917 (+ 12091us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:49.643977 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:49.646544 (+ 2567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:49.656004 (+ 9460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:49.656019 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:49.657124 (+ 1105us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:15:49.658124 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:49.658158 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:49.660146 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:49.660151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:49.662711 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:49.662734 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:49.664611 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:49.664621 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:49.664974 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:49.665825 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:15:49.665859 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:15:49.668890 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:15:49.673399 (+ 4509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:15:49.673411 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:15:49.673433 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:15:49.674093 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:15:49.674588 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:15:49.674595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:15:49.674602 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:15:49.674904 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:15:49.675680 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:15:49.675690 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:15:49.676110 (+ 420us) client_negotiation.cc:769] Sending connection context
0925 09:15:49.676428 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 09:15:49.677001 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11750,"thread_start_us":11524,"threads_started":1}
I20260925 09:15:49.679455 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:49.631228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:33595 (local address 127.3.19.126:33297)
0925 09:15:49.652625 (+ 21397us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:49.652633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:49.652692 (+ 59us) server_negotiation.cc:407] Connection header received
0925 09:15:49.652796 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:49.652801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:49.652912 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:49.653078 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:49.658354 (+ 5276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:49.659952 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:49.662893 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:49.663677 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:49.663815 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:49.669141 (+ 5326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:49.669149 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:49.669153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:49.669195 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:49.673201 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:49.674269 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:49.674277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:49.674283 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:49.674442 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:49.675055 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:49.675063 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:49.675069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:49.675498 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:49.678593 (+ 3095us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:49.678755 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 09:15:49.679007 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21205,"thread_start_us":20923,"threads_started":1}
I20260925 09:15:49.681365 14772 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33297
I20260925 09:15:49.681751 14772 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
I20260925 09:15:49.682590 14772 heartbeater.cc:511] Master 127.3.19.126:33297 requested a full tablet report, sending...
I20260925 09:15:49.685169 14559 ts_manager.cc:194] Registered new tserver with Master: a5719cefab6b4f18bb8d162ac22774ed (127.3.19.65:38147)
I20260925 09:15:49.687388 14559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:33595
Sep 25 09:15:49 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327749, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:15:49.721570 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.19.84:59947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33297
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 09:15:49 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327749, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
W20260925 09:15:50.190349 14778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:15:50.190769 14778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:15:50.190955 14778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:15:50.202399 14778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:15:50.202538 14778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:15:50.202721 14778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:15:50.218086 14778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.19.84:59947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:15:50.219827 14778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:15:50.221779 14778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:15:50.240415 14785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:50.240520 14786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:50.248767 14788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:15:50.252939 14778 server_base.cc:1034] running on GCE node
I20260925 09:15:50.253882 14778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:15:50.257287 14778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:15:50.258546 14778 hybrid_clock.cc:648] HybridClock initialized: now 1790327750258407 us; error 129 us; skew 500 ppm
I20260925 09:15:50.258956 14778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:15:50 dist-test-slave-l6qh krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327750, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:15:50.263206 14778 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:15:50.265952 14778 webserver.cc:460] Webserver started at http://127.3.19.66:43383/ using document root <none> and password file <none>
I20260925 09:15:50.266719 14778 fs_manager.cc:359] Metadata directory not provided
I20260925 09:15:50.266860 14778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:15:50.267208 14778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 09:15:50 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327750, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:15:50.287288 14778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "449dbd684db34a859c22063504b37525"
format_stamp: "Formatted at 2026-09-25 09:15:50 on dist-test-slave-l6qh"
server_key: "1397ded051311b174e1afe2dde4e3685"
server_key_iv: "d78afe32ef44e1aa24c1303dc31e7fa6"
server_key_version: "kuduclusterkey@0"
I20260925 09:15:50.288442 14778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "449dbd684db34a859c22063504b37525"
format_stamp: "Formatted at 2026-09-25 09:15:50 on dist-test-slave-l6qh"
server_key: "1397ded051311b174e1afe2dde4e3685"
server_key_iv: "d78afe32ef44e1aa24c1303dc31e7fa6"
server_key_version: "kuduclusterkey@0"
I20260925 09:15:50.295730 14778 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260925 09:15:50.314064 14802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:50.316133 14778 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 09:15:50.316303 14778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "449dbd684db34a859c22063504b37525"
format_stamp: "Formatted at 2026-09-25 09:15:50 on dist-test-slave-l6qh"
server_key: "1397ded051311b174e1afe2dde4e3685"
server_key_iv: "d78afe32ef44e1aa24c1303dc31e7fa6"
server_key_version: "kuduclusterkey@0"
I20260925 09:15:50.316546 14778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:15:50.353075 14778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:15:50.358299 14778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:15:50.359036 14778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:15:50.360175 14778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:15:50.361959 14778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:15:50.362049 14778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:50.362154 14778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:15:50.362210 14778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:50.406267 14778 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:41241
I20260925 09:15:50.406286 14915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:41241 every 8 connection(s)
I20260925 09:15:50.407820 14778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:15:50.412274 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 14778
I20260925 09:15:50.412451 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:15:50.412966 3149 external_mini_cluster.cc:1428] Setting key 39bdf4fa7b1b313d6430d407f4641caf
Sep 25 09:15:50 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327750, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:15:50.433163 14918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:50.411156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:35233 (local address 127.3.19.126:33297)
0925 09:15:50.411627 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:50.411643 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:50.415448 (+ 3805us) server_negotiation.cc:407] Connection header received
0925 09:15:50.415524 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:50.415531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:50.415641 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:50.415745 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:50.417751 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:50.418883 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:50.421554 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:50.422229 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:50.422353 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:50.426844 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:50.426856 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:50.426860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:50.426894 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:50.429819 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:50.430819 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:50.430824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:50.430827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:50.431058 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:50.431524 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:50.431529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:50.431532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:50.431797 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:50.431948 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:50.432581 (+ 633us) server_negotiation.cc:299] Negotiation successful
0925 09:15:50.432798 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":223,"threads_started":1}
I20260925 09:15:50.434175 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:50.411711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33297 (local address 127.3.19.66:35233)
0925 09:15:50.412719 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:50.412765 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:50.415109 (+ 2344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:50.416035 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:50.416049 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:50.416683 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:15:50.417564 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:50.417592 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:50.419046 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:50.419054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:50.421408 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:50.421425 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:50.422440 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:50.422449 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:50.422824 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:50.424083 (+ 1259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:15:50.424114 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:15:50.426624 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:15:50.429971 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:15:50.429980 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:15:50.429995 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:15:50.430668 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:15:50.431185 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:15:50.431190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:15:50.431193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:15:50.431401 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:15:50.431919 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:15:50.431926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:15:50.432234 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 09:15:50.432484 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 09:15:50.432969 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":412,"threads_started":1}
I20260925 09:15:50.435585 14916 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33297
I20260925 09:15:50.436064 14916 heartbeater.cc:464] Registering TS with master...
I20260925 09:15:50.436777 14916 heartbeater.cc:511] Master 127.3.19.126:33297 requested a full tablet report, sending...
I20260925 09:15:50.438311 14559 ts_manager.cc:194] Registered new tserver with Master: 449dbd684db34a859c22063504b37525 (127.3.19.66:41241)
I20260925 09:15:50.439098 14559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:35233
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:15:50.474675 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.19.84:59947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33297
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:15:50.691913 14772 heartbeater.cc:503] Master 127.3.19.126:33297 was elected leader, sending a full tablet report...
W20260925 09:15:50.854585 14922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:15:50.854874 14922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:15:50.855033 14922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:15:50.862023 14922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:15:50.862179 14922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:15:50.862340 14922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:15:50.871665 14922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.19.84:59947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:15:50.872926 14922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:15:50.874166 14922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:15:50.886704 14927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:50.886662 14930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:15:50.886632 14928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:15:50.887912 14922 server_base.cc:1034] running on GCE node
I20260925 09:15:50.888399 14922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:15:50.889204 14922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:15:50.890462 14922 hybrid_clock.cc:648] HybridClock initialized: now 1790327750890405 us; error 35 us; skew 500 ppm
I20260925 09:15:50.890765 14922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:15:50 dist-test-slave-l6qh krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327750, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:15:50.893842 14922 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:15:50.895201 14922 webserver.cc:460] Webserver started at http://127.3.19.67:45689/ using document root <none> and password file <none>
I20260925 09:15:50.895813 14922 fs_manager.cc:359] Metadata directory not provided
I20260925 09:15:50.895903 14922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:15:50.896134 14922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 09:15:50 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327750, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:15:50.911617 14922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "0931abb9cabc48f0bf1f776138dff510"
format_stamp: "Formatted at 2026-09-25 09:15:50 on dist-test-slave-l6qh"
server_key: "be24092dcf6ced61bcbdf883fe274d73"
server_key_iv: "45f734922e0c43521585f79035348328"
server_key_version: "kuduclusterkey@0"
I20260925 09:15:50.912667 14922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "0931abb9cabc48f0bf1f776138dff510"
format_stamp: "Formatted at 2026-09-25 09:15:50 on dist-test-slave-l6qh"
server_key: "be24092dcf6ced61bcbdf883fe274d73"
server_key_iv: "45f734922e0c43521585f79035348328"
server_key_version: "kuduclusterkey@0"
I20260925 09:15:50.918478 14922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 09:15:50.933887 14937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:50.935382 14922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 09:15:50.935565 14922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "0931abb9cabc48f0bf1f776138dff510"
format_stamp: "Formatted at 2026-09-25 09:15:50 on dist-test-slave-l6qh"
server_key: "be24092dcf6ced61bcbdf883fe274d73"
server_key_iv: "45f734922e0c43521585f79035348328"
server_key_version: "kuduclusterkey@0"
I20260925 09:15:50.935786 14922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:15:50.965541 14922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:15:50.969050 14922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:15:50.969401 14922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:15:50.970395 14922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:15:50.971673 14922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:15:50.971745 14922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:50.971912 14922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:15:50.971966 14922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:15:51.009938 14922 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:41183
I20260925 09:15:51.009971 15050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:41183 every 8 connection(s)
I20260925 09:15:51.011193 14922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:15:51.021292 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 14922
I20260925 09:15:51.021476 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:15:51.021942 3149 external_mini_cluster.cc:1428] Setting key 940e2307e546c74b9697d2a9d40d6759
Sep 25 09:15:51 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327750, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:15:51.030073 15053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.013683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:45847 (local address 127.3.19.126:33297)
0925 09:15:51.014184 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:51.014188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:51.014972 (+ 784us) server_negotiation.cc:407] Connection header received
0925 09:15:51.016636 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:51.016642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:51.016747 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:51.016867 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:51.018287 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.018996 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.020538 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.021001 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.021132 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.025272 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:51.025282 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:51.025285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:51.025310 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:51.027636 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:51.028260 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:51.028263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:51.028265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:51.028371 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:51.028731 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:51.028734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:51.028736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:51.028968 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:51.029076 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:51.029694 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 09:15:51.029843 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":275,"threads_started":1}
I20260925 09:15:51.030922 15054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.014143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33297 (local address 127.3.19.67:45847)
0925 09:15:51.014802 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:51.014843 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:51.016355 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:51.017092 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:51.017099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:51.017583 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:15:51.018121 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:51.018137 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.019143 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.019155 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:51.020374 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:51.020386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.021256 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.021266 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:51.021686 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.022350 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:15:51.022370 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:15:51.025070 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:15:51.027754 (+ 2684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:15:51.027760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:15:51.027770 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:15:51.028157 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:15:51.028480 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:15:51.028483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:15:51.028485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:15:51.028636 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:15:51.029062 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:15:51.029067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:15:51.029390 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 09:15:51.029590 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 09:15:51.029930 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":268,"threads_started":1}
I20260925 09:15:51.032243 15051 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33297
I20260925 09:15:51.032506 15051 heartbeater.cc:464] Registering TS with master...
I20260925 09:15:51.033116 15051 heartbeater.cc:511] Master 127.3.19.126:33297 requested a full tablet report, sending...
I20260925 09:15:51.034449 14559 ts_manager.cc:194] Registered new tserver with Master: 0931abb9cabc48f0bf1f776138dff510 (127.3.19.67:41183)
I20260925 09:15:51.035254 14559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:45847
I20260925 09:15:51.037272 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:15:51.054153 15053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.041970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.3.19.126:33297)
0925 09:15:51.042178 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:51.042188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:51.042584 (+ 396us) server_negotiation.cc:407] Connection header received
0925 09:15:51.042747 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:51.042750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:51.042814 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:51.042888 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:51.044262 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.045054 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.046862 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.047240 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.047328 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.049571 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:51.049576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:51.049579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:51.049605 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:51.051914 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:51.052540 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:51.052544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:51.052546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:51.052655 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:51.053004 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:51.053007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:51.053008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:51.053287 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:51.053448 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:51.053718 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 09:15:51.053863 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 09:15:51.057830 14559 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:15:51.064795 14559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:15:51.097631 15066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.077023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.3.19.65:38147)
0925 09:15:51.089043 (+ 12020us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:51.089050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:51.089074 (+ 24us) server_negotiation.cc:407] Connection header received
0925 09:15:51.089160 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:51.089168 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:51.089428 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:51.089623 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:15:51.090374 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.092495 (+ 2121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.095174 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.096433 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.096559 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.096779 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:51.097072 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 09:15:51.097292 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11875,"thread_start_us":11755,"threads_started":1}
I20260925 09:15:51.097651 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.076862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:38147 (local address 127.0.0.1:45628)
0925 09:15:51.087048 (+ 10186us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:51.087086 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:51.087260 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:51.089645 (+ 2385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:51.089652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:51.089679 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:15:51.090133 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:51.090149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.092656 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.092668 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:51.095022 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:51.095033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.096620 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.096626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:51.096817 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.096873 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 09:15:51.097012 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 09:15:51.097181 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9955,"mutex_wait_us":61,"thread_start_us":13255,"threads_started":1}
I20260925 09:15:51.098306 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.077859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.3.19.67:41183)
0925 09:15:51.088607 (+ 10748us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:51.088613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:51.090505 (+ 1892us) server_negotiation.cc:407] Connection header received
0925 09:15:51.090813 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:51.090820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:51.091034 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:51.091187 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:15:51.091976 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.093292 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.095872 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.096943 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.097049 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.097284 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:51.097846 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 09:15:51.097990 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10609,"thread_start_us":10466,"threads_started":1}
I20260925 09:15:51.098618 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.076952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:41241 (local address 127.0.0.1:43902)
0925 09:15:51.087663 (+ 10711us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:51.087683 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:51.087783 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:51.090500 (+ 2717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:51.090507 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:51.090534 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:15:51.091427 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:51.091440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.093100 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.093107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:51.095667 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:51.095682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.097580 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.097586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:51.097868 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.097891 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 09:15:51.097995 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 09:15:51.098107 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10592,"mutex_wait_us":185,"thread_start_us":10487,"threads_started":1}
I20260925 09:15:51.099301 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.077273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.3.19.66:41241)
0925 09:15:51.089441 (+ 12168us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:51.089454 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:51.089490 (+ 36us) server_negotiation.cc:407] Connection header received
0925 09:15:51.089623 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:51.089635 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:51.089887 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:51.090273 (+ 386us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:15:51.091620 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.092977 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.095837 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.097115 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.097231 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.097423 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:51.098772 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0925 09:15:51.098935 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12000,"thread_start_us":11870,"threads_started":1}
I20260925 09:15:51.099409 15064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.077627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:41183 (local address 127.0.0.1:52996)
0925 09:15:51.090404 (+ 12777us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:51.090422 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:51.090603 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:51.091501 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:51.091504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:51.091522 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:15:51.091796 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:51.091803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.093510 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.093515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:51.095715 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:51.095724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.097411 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.097417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:51.097582 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.097653 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 09:15:51.098476 (+ 823us) client_negotiation.cc:240] Negotiation successful
0925 09:15:51.098611 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12636,"mutex_wait_us":22,"thread_start_us":9031,"threads_started":1}
I20260925 09:15:51.102690 14851 tablet_service.cc:1467] Processing CreateTablet for tablet 4c1bfe97bfac46388f4ccd2cbca70cf7 (DEFAULT_TABLE table=test-table [id=06220d1e0fdd414e940f00bb02c1eaac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:15:51.102701 14707 tablet_service.cc:1467] Processing CreateTablet for tablet 4c1bfe97bfac46388f4ccd2cbca70cf7 (DEFAULT_TABLE table=test-table [id=06220d1e0fdd414e940f00bb02c1eaac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:15:51.103053 14985 tablet_service.cc:1467] Processing CreateTablet for tablet 4c1bfe97bfac46388f4ccd2cbca70cf7 (DEFAULT_TABLE table=test-table [id=06220d1e0fdd414e940f00bb02c1eaac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:15:51.104197 14707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c1bfe97bfac46388f4ccd2cbca70cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:15:51.104301 14851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c1bfe97bfac46388f4ccd2cbca70cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:15:51.104310 14985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c1bfe97bfac46388f4ccd2cbca70cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:15:51.114778 15070 tablet_bootstrap.cc:492] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510: Bootstrap starting.
I20260925 09:15:51.118450 15071 tablet_bootstrap.cc:492] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed: Bootstrap starting.
I20260925 09:15:51.119004 15070 tablet_bootstrap.cc:654] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510: Neither blocks nor log segments found. Creating new log.
I20260925 09:15:51.120309 15070 log.cc:824] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510: Log is configured to *not* fsync() on all Append() calls
I20260925 09:15:51.121285 15072 tablet_bootstrap.cc:492] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525: Bootstrap starting.
I20260925 09:15:51.122429 15071 tablet_bootstrap.cc:654] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed: Neither blocks nor log segments found. Creating new log.
I20260925 09:15:51.123821 15071 log.cc:824] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed: Log is configured to *not* fsync() on all Append() calls
I20260925 09:15:51.124053 15072 tablet_bootstrap.cc:654] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525: Neither blocks nor log segments found. Creating new log.
I20260925 09:15:51.125144 15072 log.cc:824] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525: Log is configured to *not* fsync() on all Append() calls
I20260925 09:15:51.126925 15071 tablet_bootstrap.cc:492] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed: No bootstrap required, opened a new log
I20260925 09:15:51.127187 15071 ts_tablet_manager.cc:1397] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20260925 09:15:51.127803 15070 tablet_bootstrap.cc:492] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510: No bootstrap required, opened a new log
I20260925 09:15:51.128022 15070 ts_tablet_manager.cc:1397] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260925 09:15:51.128396 15072 tablet_bootstrap.cc:492] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525: No bootstrap required, opened a new log
I20260925 09:15:51.128610 15072 ts_tablet_manager.cc:1397] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260925 09:15:51.134943 15072 raft_consensus.cc:348] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5719cefab6b4f18bb8d162ac22774ed" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 38147 } } peers { permanent_uuid: "449dbd684db34a859c22063504b37525" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 41241 } } peers { permanent_uuid: "0931abb9cabc48f0bf1f776138dff510" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41183 } }
I20260925 09:15:51.135260 15072 raft_consensus.cc:374] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:15:51.135370 15072 raft_consensus.cc:729] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449dbd684db34a859c22063504b37525, State: Initialized, Role: FOLLOWER
I20260925 09:15:51.136018 15072 consensus_queue.cc:260] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5719cefab6b4f18bb8d162ac22774ed" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 38147 } } peers { permanent_uuid: "449dbd684db34a859c22063504b37525" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 41241 } } peers { permanent_uuid: "0931abb9cabc48f0bf1f776138dff510" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41183 } }
I20260925 09:15:51.138010 14916 heartbeater.cc:503] Master 127.3.19.126:33297 was elected leader, sending a full tablet report...
I20260925 09:15:51.137791 15071 raft_consensus.cc:348] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5719cefab6b4f18bb8d162ac22774ed" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 38147 } } peers { permanent_uuid: "449dbd684db34a859c22063504b37525" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 41241 } } peers { permanent_uuid: "0931abb9cabc48f0bf1f776138dff510" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41183 } }
I20260925 09:15:51.138150 15071 raft_consensus.cc:374] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:15:51.138242 15071 raft_consensus.cc:729] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5719cefab6b4f18bb8d162ac22774ed, State: Initialized, Role: FOLLOWER
I20260925 09:15:51.138733 15072 ts_tablet_manager.cc:1428] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525: Time spent starting tablet: real 0.010s user 0.007s sys 0.005s
I20260925 09:15:51.138572 15070 raft_consensus.cc:348] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5719cefab6b4f18bb8d162ac22774ed" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 38147 } } peers { permanent_uuid: "449dbd684db34a859c22063504b37525" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 41241 } } peers { permanent_uuid: "0931abb9cabc48f0bf1f776138dff510" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41183 } }
I20260925 09:15:51.138926 15070 raft_consensus.cc:374] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:15:51.139017 15070 raft_consensus.cc:729] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0931abb9cabc48f0bf1f776138dff510, State: Initialized, Role: FOLLOWER
I20260925 09:15:51.138878 15071 consensus_queue.cc:260] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5719cefab6b4f18bb8d162ac22774ed" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 38147 } } peers { permanent_uuid: "449dbd684db34a859c22063504b37525" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 41241 } } peers { permanent_uuid: "0931abb9cabc48f0bf1f776138dff510" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41183 } }
I20260925 09:15:51.139808 15070 consensus_queue.cc:260] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5719cefab6b4f18bb8d162ac22774ed" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 38147 } } peers { permanent_uuid: "449dbd684db34a859c22063504b37525" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 41241 } } peers { permanent_uuid: "0931abb9cabc48f0bf1f776138dff510" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41183 } }
I20260925 09:15:51.140959 15051 heartbeater.cc:503] Master 127.3.19.126:33297 was elected leader, sending a full tablet report...
I20260925 09:15:51.141920 15070 ts_tablet_manager.cc:1428] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260925 09:15:51.143581 15071 ts_tablet_manager.cc:1428] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
W20260925 09:15:51.157135 14773 tablet.cc:2369] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:15:51.160573 14917 tablet.cc:2369] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:15:51.229130 15078 raft_consensus.cc:482] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:15:51.229449 15078 raft_consensus.cc:504] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5719cefab6b4f18bb8d162ac22774ed" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 38147 } } peers { permanent_uuid: "449dbd684db34a859c22063504b37525" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 41241 } } peers { permanent_uuid: "0931abb9cabc48f0bf1f776138dff510" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41183 } }
I20260925 09:15:51.231156 15078 leader_election.cc:288] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5719cefab6b4f18bb8d162ac22774ed (127.3.19.65:38147), 449dbd684db34a859c22063504b37525 (127.3.19.66:41241)
I20260925 09:15:51.237422 15054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.231513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:38147 (local address 127.3.19.67:39583)
0925 09:15:51.231834 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:51.231849 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:51.231972 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:51.232429 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:51.232435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:51.232468 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:15:51.232677 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:51.232689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.233661 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.233667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:51.235580 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:51.235592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.236834 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.236838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:51.236954 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.236977 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 09:15:51.237053 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 09:15:51.237138 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260925 09:15:51.237491 15066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.231656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:39583 (local address 127.3.19.65:38147)
0925 09:15:51.231906 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:51.231914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:51.231940 (+ 26us) server_negotiation.cc:407] Connection header received
0925 09:15:51.232151 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:51.232156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:51.232268 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:51.232387 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:15:51.232826 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.233539 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.235728 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.236686 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.236813 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.236913 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:51.237098 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 09:15:51.237259 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 09:15:51.238497 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.231579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:52625 (local address 127.3.19.66:41241)
0925 09:15:51.231782 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:51.231788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:51.232503 (+ 715us) server_negotiation.cc:407] Connection header received
0925 09:15:51.232746 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:51.232750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:51.232825 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:51.232924 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:15:51.233466 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.234278 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.236443 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.237607 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.237748 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.237851 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:51.238127 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 09:15:51.238251 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 09:15:51.238730 14727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1bfe97bfac46388f4ccd2cbca70cf7" candidate_uuid: "0931abb9cabc48f0bf1f776138dff510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5719cefab6b4f18bb8d162ac22774ed" is_pre_election: true
I20260925 09:15:51.238981 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.231674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:41241 (local address 127.3.19.67:52625)
0925 09:15:51.232419 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 09:15:51.232434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:15:51.232596 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:15:51.232940 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:15:51.232944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:15:51.232969 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:15:51.233327 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:51.233336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.234448 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.234454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:51.236258 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:15:51.236270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.237858 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.237866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:15:51.237987 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.238009 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 09:15:51.238105 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 09:15:51.238735 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":385,"threads_started":1}
I20260925 09:15:51.239228 14727 raft_consensus.cc:2454] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0931abb9cabc48f0bf1f776138dff510 in term 0.
I20260925 09:15:51.239957 14941 leader_election.cc:302] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0931abb9cabc48f0bf1f776138dff510, a5719cefab6b4f18bb8d162ac22774ed; no voters:
I20260925 09:15:51.239939 14871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1bfe97bfac46388f4ccd2cbca70cf7" candidate_uuid: "0931abb9cabc48f0bf1f776138dff510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449dbd684db34a859c22063504b37525" is_pre_election: true
I20260925 09:15:51.240351 15078 raft_consensus.cc:2781] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:15:51.240477 15078 raft_consensus.cc:482] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:15:51.240592 15078 raft_consensus.cc:3037] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:15:51.240504 14871 raft_consensus.cc:2454] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0931abb9cabc48f0bf1f776138dff510 in term 0.
I20260925 09:15:51.243016 15078 raft_consensus.cc:504] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5719cefab6b4f18bb8d162ac22774ed" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 38147 } } peers { permanent_uuid: "449dbd684db34a859c22063504b37525" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 41241 } } peers { permanent_uuid: "0931abb9cabc48f0bf1f776138dff510" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41183 } }
I20260925 09:15:51.243593 15078 leader_election.cc:288] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [CANDIDATE]: Term 1 election: Requested vote from peers a5719cefab6b4f18bb8d162ac22774ed (127.3.19.65:38147), 449dbd684db34a859c22063504b37525 (127.3.19.66:41241)
I20260925 09:15:51.243974 14727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1bfe97bfac46388f4ccd2cbca70cf7" candidate_uuid: "0931abb9cabc48f0bf1f776138dff510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5719cefab6b4f18bb8d162ac22774ed"
I20260925 09:15:51.244061 14871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1bfe97bfac46388f4ccd2cbca70cf7" candidate_uuid: "0931abb9cabc48f0bf1f776138dff510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449dbd684db34a859c22063504b37525"
I20260925 09:15:51.244215 14871 raft_consensus.cc:3037] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:15:51.244189 14727 raft_consensus.cc:3037] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:15:51.247382 14871 raft_consensus.cc:2454] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0931abb9cabc48f0bf1f776138dff510 in term 1.
I20260925 09:15:51.247382 14727 raft_consensus.cc:2454] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0931abb9cabc48f0bf1f776138dff510 in term 1.
I20260925 09:15:51.247763 14941 leader_election.cc:302] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0931abb9cabc48f0bf1f776138dff510, a5719cefab6b4f18bb8d162ac22774ed; no voters:
I20260925 09:15:51.248054 15078 raft_consensus.cc:2781] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:15:51.248495 15078 raft_consensus.cc:686] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [term 1 LEADER]: Becoming Leader. State: Replica: 0931abb9cabc48f0bf1f776138dff510, State: Running, Role: LEADER
I20260925 09:15:51.249030 15078 consensus_queue.cc:237] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5719cefab6b4f18bb8d162ac22774ed" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 38147 } } peers { permanent_uuid: "449dbd684db34a859c22063504b37525" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 41241 } } peers { permanent_uuid: "0931abb9cabc48f0bf1f776138dff510" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41183 } }
I20260925 09:15:51.253474 14558 catalog_manager.cc:5495] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0931abb9cabc48f0bf1f776138dff510 (127.3.19.67). New cstate: current_term: 1 leader_uuid: "0931abb9cabc48f0bf1f776138dff510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5719cefab6b4f18bb8d162ac22774ed" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449dbd684db34a859c22063504b37525" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 41241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0931abb9cabc48f0bf1f776138dff510" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41183 } health_report { overall_health: HEALTHY } } }
W20260925 09:15:51.262713 15052 tablet.cc:2369] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:15:51.287470 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.282061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.3.19.67:41183)
0925 09:15:51.282272 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:51.282278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:51.282369 (+ 91us) server_negotiation.cc:407] Connection header received
0925 09:15:51.282592 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:51.282596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:51.282680 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:51.282823 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:15:51.283195 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.283859 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.285340 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.285726 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.285877 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.286367 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:15:51.286892 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:15:51.287065 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:51.287121 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 09:15:51.287203 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 09:15:51.295346 14871 raft_consensus.cc:1261] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525 [term 1 FOLLOWER]: Refusing update from remote peer 0931abb9cabc48f0bf1f776138dff510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:15:51.295424 14727 raft_consensus.cc:1261] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed [term 1 FOLLOWER]: Refusing update from remote peer 0931abb9cabc48f0bf1f776138dff510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:15:51.296296 15078 consensus_queue.cc:1035] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449dbd684db34a859c22063504b37525" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 41241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:15:51.296564 15080 consensus_queue.cc:1035] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5719cefab6b4f18bb8d162ac22774ed" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:15:51.321065 14559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:15:51.321355 14559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:15:51.324590 14559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b80f833f097e469aaa581a1e034079b8: Sending DeleteTablet for 3 replicas of tablet 4c1bfe97bfac46388f4ccd2cbca70cf7
I20260925 09:15:51.325610 14707 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c1bfe97bfac46388f4ccd2cbca70cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:15:51 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:45628
I20260925 09:15:51.325680 14851 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c1bfe97bfac46388f4ccd2cbca70cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:15:51 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:43902
I20260925 09:15:51.325693 14985 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c1bfe97bfac46388f4ccd2cbca70cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:15:51 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:52996
I20260925 09:15:51.332867 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 14639
I20260925 09:15:51.335872 15095 tablet_replica.cc:330] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510: stopping tablet replica
I20260925 09:15:51.336267 15095 raft_consensus.cc:2229] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:15:51.336925 15095 raft_consensus.cc:2258] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:15:51.338865 15095 ts_tablet_manager.cc:1905] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:15:51.339965 15094 tablet_replica.cc:330] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed: stopping tablet replica
I20260925 09:15:51.340322 15094 raft_consensus.cc:2229] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:15:51.340788 15094 raft_consensus.cc:2258] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:15:51.346429 15094 ts_tablet_manager.cc:1905] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:15:51.353696 15093 tablet_replica.cc:330] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525: stopping tablet replica
I20260925 09:15:51.354121 15093 raft_consensus.cc:2229] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525 [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 25 09:15:51 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:15:51.354705 15093 raft_consensus.cc:2258] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:15:51.356690 15093 ts_tablet_manager.cc:1905] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:15:51.357969 15095 ts_tablet_manager.cc:1918] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:15:51.358125 15095 log.cc:1196] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/4c1bfe97bfac46388f4ccd2cbca70cf7
I20260925 09:15:51.358712 15095 ts_tablet_manager.cc:1939] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 0931abb9cabc48f0bf1f776138dff510: Deleting consensus metadata
I20260925 09:15:51.360453 14543 catalog_manager.cc:4841] TS 0931abb9cabc48f0bf1f776138dff510 (127.3.19.67:41183): tablet 4c1bfe97bfac46388f4ccd2cbca70cf7 (table test-table [id=06220d1e0fdd414e940f00bb02c1eaac]) successfully deleted
I20260925 09:15:51.361127 15094 ts_tablet_manager.cc:1918] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:15:51.361406 15094 log.cc:1196] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/4c1bfe97bfac46388f4ccd2cbca70cf7
I20260925 09:15:51.362082 15094 ts_tablet_manager.cc:1939] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P a5719cefab6b4f18bb8d162ac22774ed: Deleting consensus metadata
I20260925 09:15:51.363003 15066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.334592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.3.19.65:38147)
0925 09:15:51.334800 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:51.334809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:51.341697 (+ 6888us) server_negotiation.cc:407] Connection header received
0925 09:15:51.341864 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:51.341874 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:51.341974 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:51.342100 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:51.344405 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.345914 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.348505 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.349373 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.349496 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.355618 (+ 6122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:51.355631 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:51.355646 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:51.355709 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:51.359279 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:51.360168 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:51.360176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:51.360179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:51.360333 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:51.360828 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:51.360834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:51.360837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:51.361452 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:51.361729 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:51.362318 (+ 589us) server_negotiation.cc:299] Negotiation successful
0925 09:15:51.362568 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 09:15:51.364027 15093 ts_tablet_manager.cc:1918] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:15:51.364188 15093 log.cc:1196] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/4c1bfe97bfac46388f4ccd2cbca70cf7
I20260925 09:15:51.364188 14545 catalog_manager.cc:4841] TS a5719cefab6b4f18bb8d162ac22774ed (127.3.19.65:38147): tablet 4c1bfe97bfac46388f4ccd2cbca70cf7 (table test-table [id=06220d1e0fdd414e940f00bb02c1eaac]) successfully deleted
I20260925 09:15:51.364369 14767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:15:51.364573 15093 ts_tablet_manager.cc:1939] T 4c1bfe97bfac46388f4ccd2cbca70cf7 P 449dbd684db34a859c22063504b37525: Deleting consensus metadata
I20260925 09:15:51.366150 14542 catalog_manager.cc:4841] TS 449dbd684db34a859c22063504b37525 (127.3.19.66:41241): tablet 4c1bfe97bfac46388f4ccd2cbca70cf7 (table test-table [id=06220d1e0fdd414e940f00bb02c1eaac]) successfully deleted
I20260925 09:15:51.451576 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 14639
I20260925 09:15:51.470346 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 14778
Sep 25 09:15:51 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:15:51.487412 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.471226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.3.19.66:41241)
0925 09:15:51.471395 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:51.471399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:51.471417 (+ 18us) server_negotiation.cc:407] Connection header received
0925 09:15:51.471538 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:51.471542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:51.471615 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:51.471716 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:51.473450 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.474133 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.476048 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.476430 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.476522 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.481627 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:51.481641 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:51.481656 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:51.481704 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:51.484691 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:51.485365 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:51.485380 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:51.485383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:51.485523 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:51.485923 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:51.485933 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:51.485935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:51.486281 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:51.486567 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:51.486864 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 09:15:51.487062 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 09:15:51.488092 14911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:15:51.578769 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 14778
I20260925 09:15:51.596490 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 14922
Sep 25 09:15:51 dist-test-slave-l6qh krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:15:51.612684 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:15:51.597339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53004 (local address 127.3.19.67:41183)
0925 09:15:51.597513 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 09:15:51.597539 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 09:15:51.597619 (+ 80us) server_negotiation.cc:407] Connection header received
0925 09:15:51.597819 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:15:51.597824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:15:51.597895 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:15:51.598006 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:15:51.599360 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.600258 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.601672 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:15:51.602084 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:15:51.602228 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:15:51.606767 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:15:51.606779 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:15:51.606793 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:15:51.606827 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:15:51.609896 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:51.610724 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:51.610730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:51.610733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:51.610858 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:15:51.611279 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:15:51.611282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:15:51.611284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:15:51.611615 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:15:51.611871 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 09:15:51.612206 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 09:15:51.612362 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 09:15:51.613453 15045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:15:51.737072 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 14922
I20260925 09:15:51.753176 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 14526
I20260925 09:15:51.754614 14589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:15:51.868152 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 14526
I20260925 09:15:51.892593 3149 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 09:15:52.227902 3149 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 09:15:52.228092 3149 mini_ranger.cc:66] Stopping Ranger...
I20260925 09:15:52.573444 3149 mini_ranger.cc:68] Stopped Ranger
2026-09-25 09:15:52.573 UTC [14109] LOG: received smart shutdown request
2026-09-25 09:15:52.577 UTC [14109] LOG: background worker "logical replication launcher" (PID 14118) exited with exit code 1
2026-09-25 09:15:52.577 UTC [14113] LOG: shutting down
2026-09-25 09:15:52.733 UTC [14109] LOG: database system is shut down
2026-09-25T09:15:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49321 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:15:52 dist-test-slave-l6qh krb5kdc[15109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:15:52 dist-test-slave-l6qh krb5kdc[15109](info): set up 2 sockets
Sep 25 09:15:52 dist-test-slave-l6qh krb5kdc[15109](info): commencing operation
krb5kdc: starting...
W20260925 09:15:53.942437 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:15:54 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:15:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:15:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/rangeradmin_127.3.19.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/rangeradmin_127.3.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/rangerlookup_127.3.19.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/rangerlookup_127.3.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/HTTP_127.3.19.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/HTTP_127.3.19.84@KRBTEST.COM.keytab.
I20260925 09:15:54.244244 3149 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCZfsc6/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/postgres -l logfile start
2026-09-25 09:15:56.629 UTC [15140] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 09:15:56.629 UTC [15140] LOG: listening on IPv4 address "127.3.19.84", port 58711
2026-09-25 09:15:56.632 UTC [15140] LOG: listening on Unix socket "/tmp/.s.PGSQL.58711"
2026-09-25 09:15:56.654 UTC [15143] LOG: database system was shut down at 2026-09-25 09:15:55 UTC
2026-09-25 09:15:56.657 UTC [15140] LOG: database system is ready to accept connections
I20260925 09:15:57.606155 3149 mini_postgres.cc:96] Postgres bound to 58711
2026-09-25 09:15:57.623 UTC [15151] FATAL: database "slave" does not exist
127.3.19.84:58711 - accepting connections
I20260925 09:15:57.624428 3149 mini_ranger.cc:161] Starting Ranger...
I20260925 09:15:57.649539 3149 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 09:15:58.102200 3149 mini_ranger.cc:87] Created ranger Postgres database
I20260925 09:15:58.102329 3149 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-admin
2026-09-25 09:15:58,395 [I] DB FLAVOR :POSTGRES
2026-09-25 09:15:58,396 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 09:15:58,396 [I] Checking connection..
2026-09-25 09:15:58,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 09:15:58,629 [I] Checking connection passed.
2026-09-25 09:15:58,629 [I] --------- Verifying version history table ---------
2026-09-25 09:15:58,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 09:15:58,883 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 09:15:58,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 09:15:59,134 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 09:15:59,135 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 09:15:59,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 09:15:59.385 UTC [15216] WARNING: there is no transaction in progress
2026-09-25 09:15:59,391 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 09:15:59,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 09:15:59,612 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 09:15:59,612 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 09:15:59,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 09:15:59,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 09:16:00,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-l6qh.c.gcp-upstream.internal','N') ;"
2026-09-25 09:16:00,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 09:16:00,553 [I] Table x_portal_user does not exist in database ranger
2026-09-25 09:16:00,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 09:16:00,799 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 09:16:00,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 09:16:01,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 09:16:01,305 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 09:16:01,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 09:16:02.260 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.276 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.292 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.307 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.324 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.343 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.397 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.404 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.419 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.428 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.438 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.452 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.462 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.469 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.475 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:02.480 UTC [15324] WARNING: there is no transaction in progress
2026-09-25 09:16:03,070 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 09:16:03,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 09:16:03,320 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 09:16:03,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 09:16:03,576 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 09:16:03,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 09:16:03,808 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 09:16:03,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 09:16:04,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 09:16:04,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-l6qh.c.gcp-upstream.internal';"
2026-09-25 09:16:04,544 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 09:16:04,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 09:16:04,789 [I] DB_PATCHES have already been applied
I20260925 09:16:04.795601 3149 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCZfsc6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 09:16:04.795686 3149 mini_ranger.cc:193] Using host: 127.3.19.84
I20260925 09:16:04.811003 3149 mini_ranger.cc:227] Ranger admin URL: http://127.3.19.84:59913
Sep 25, 2026 9:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 9:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 9:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 9:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 9:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 9:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 9:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 09:16:05.935617 3149 mini_ranger.cc:160] Time spent starting Ranger: real 8.311s user 0.000s sys 0.041s
Sep 25, 2026 9:16:25 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 9:16:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 9:16:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 9:16:25 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 9:16:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 9:16:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 9:16:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 9:16:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 9:16:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 9:16:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 9:16:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 9:16:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 9:16:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 9:16:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 9:16:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 9:16:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 9:16:25 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 9:16:26 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 09:16:27.406828 3149 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/rangerkms_127.3.19.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/rangerkms_127.3.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.19.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.19.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.19.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/HTTP_127.3.19.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.19.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/HTTP_127.3.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 09:16:27 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327787, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 09:16:27.593537 3149 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 09:16:27.615891 3149 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 09:16:28.419009 3149 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 09:16:28.419131 3149 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-kms
I20260925 09:16:28.419176 3149 mini_ranger_kms.cc:227] Using postgres at 127.3.19.84:58711
2026-09-25 09:16:28,764 [I] DB FLAVOR :POSTGRES
2026-09-25 09:16:28,765 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 09:16:28,765 [I] Checking connection
2026-09-25 09:16:28,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 09:16:28,996 [I] connection success
2026-09-25 09:16:28,997 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 09:16:28,997 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 09:16:28,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 09:16:29,241 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 09:16:29,241 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 09:16:29,242 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 09:16:29,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.19.84:58711/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 09:16:29,486 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 09:16:29.948997 3149 mini_ranger_kms.cc:310] Created kms service
I20260925 09:16:30.158967 3149 mini_ranger_kms.cc:326] Created kudu user
I20260925 09:16:30.203795 3149 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 09:16:30 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327787, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:16:30.577435 3149 mini_ranger_kms.cc:384] Added ranger policy
I20260925 09:16:30.577805 3149 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCZfsc6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCZfsc6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 09:16:30.577888 3149 mini_ranger_kms.cc:240] Using host: 127.3.19.84
I20260925 09:16:30.597748 3149 mini_ranger_kms.cc:276] Ranger KMS PID: 15506
I20260925 09:16:30.597877 3149 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.19.84:52457
Sep 25, 2026 9:16:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 9:16:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 9:16:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 9:16:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 9:16:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 9:16:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 09:16:32.897178 3149 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.304s user 0.082s sys 0.286s
I20260925 09:16:32.897467 3149 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 09:16:32.897630 3149 mini_ranger_kms.cc:401] 127.3.19.84:52457/kms/v1/keys
Sep 25 09:16:32 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790327792, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.19.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 09:16:33 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327792, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.19.84@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
Sep 25 09:16:33 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327792, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.19.84@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
Sep 25 09:16:35 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:16:35.944547 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46341
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:46341
--ranger_config_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.19.84:52457/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:16:36.265903 15557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:36.266172 15557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:36.266322 15557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:36.273399 15557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:16:36.273494 15557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:36.273584 15557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:36.273700 15557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:16:36.273777 15557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:16:36.283627 15557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.19.84:52457/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:46341
--ranger_config_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:36.284883 15557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:36.286250 15557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:36.298897 15562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:36.298856 15563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:36.298933 15565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:36.299755 15557 server_base.cc:1034] running on GCE node
I20260925 09:16:36.300521 15557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:36.302009 15557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:36.303270 15557 hybrid_clock.cc:648] HybridClock initialized: now 1790327796303201 us; error 44 us; skew 500 ppm
I20260925 09:16:36.303694 15557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:36 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:36.310428 15557 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:16:36.312006 15557 webserver.cc:460] Webserver started at http://127.3.19.126:46163/ using document root <none> and password file <none>
I20260925 09:16:36.312678 15557 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:36.312810 15557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:36.313109 15557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 09:16:36 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:16:36.377534 15557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "eef4711ed1144c2a8b200d496d3699b5"
format_stamp: "Formatted at 2026-09-25 09:16:36 on dist-test-slave-l6qh"
server_key: "cbb49c42b4a773e2474e8bdfc7238e8e"
server_key_iv: "3a3602aa82230c70870ce0d335f72775"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:36.378449 15557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "eef4711ed1144c2a8b200d496d3699b5"
format_stamp: "Formatted at 2026-09-25 09:16:36 on dist-test-slave-l6qh"
server_key: "cbb49c42b4a773e2474e8bdfc7238e8e"
server_key_iv: "3a3602aa82230c70870ce0d335f72775"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:36.384479 15557 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 09:16:36.404965 15572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:36.406816 15557 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 09:16:36.407008 15557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "eef4711ed1144c2a8b200d496d3699b5"
format_stamp: "Formatted at 2026-09-25 09:16:36 on dist-test-slave-l6qh"
server_key: "cbb49c42b4a773e2474e8bdfc7238e8e"
server_key_iv: "3a3602aa82230c70870ce0d335f72775"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:36.407265 15557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:36.438992 15557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:36.442608 15557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:36.442862 15557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:36.461383 15557 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:46341
I20260925 09:16:36.461377 15623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:46341 every 8 connection(s)
I20260925 09:16:36.462679 15557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:16:36.466535 15624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:36.467067 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 15557
I20260925 09:16:36.467345 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:16:36.468032 3149 external_mini_cluster.cc:1428] Setting key e19eb6689e8d59c86d64a1f5ed09a4a4
I20260925 09:16:36.480763 15624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5: Bootstrap starting.
Sep 25 09:16:36 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:36.484515 15624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:36.485715 15624 log.cc:824] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:36.489066 15624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5: No bootstrap required, opened a new log
I20260925 09:16:36.492326 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:36.469930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.3.19.126:46341)
0925 09:16:36.470721 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:36.470735 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:36.470780 (+ 45us) server_negotiation.cc:407] Connection header received
0925 09:16:36.472867 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:36.472887 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:36.473312 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:36.473634 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:36.475422 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:36.476646 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:36.478553 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:36.479170 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:36.479304 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:36.484118 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:36.484126 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:36.484139 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:36.484171 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:36.487695 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:36.488335 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:36.488344 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:36.488352 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:36.488574 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:36.488943 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:36.488949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:36.488952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:36.489702 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:36.490087 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:36.490387 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 09:16:36.490841 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":585,"thread_start_us":399,"threads_started":1}
I20260925 09:16:36.498099 15624 raft_consensus.cc:348] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef4711ed1144c2a8b200d496d3699b5" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46341 } }
I20260925 09:16:36.498486 15624 raft_consensus.cc:374] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:36.498620 15624 raft_consensus.cc:729] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef4711ed1144c2a8b200d496d3699b5, State: Initialized, Role: FOLLOWER
I20260925 09:16:36.499209 15624 consensus_queue.cc:260] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef4711ed1144c2a8b200d496d3699b5" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46341 } }
I20260925 09:16:36.499475 15624 raft_consensus.cc:388] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:16:36.499616 15624 raft_consensus.cc:482] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:16:36.499782 15624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:36.502063 15624 raft_consensus.cc:504] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef4711ed1144c2a8b200d496d3699b5" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46341 } }
I20260925 09:16:36.502569 15624 leader_election.cc:302] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eef4711ed1144c2a8b200d496d3699b5; no voters:
I20260925 09:16:36.503285 15624 leader_election.cc:288] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:16:36.503412 15629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:16:36.504197 15629 raft_consensus.cc:686] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 1 LEADER]: Becoming Leader. State: Replica: eef4711ed1144c2a8b200d496d3699b5, State: Running, Role: LEADER
I20260925 09:16:36.504750 15629 consensus_queue.cc:237] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef4711ed1144c2a8b200d496d3699b5" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46341 } }
I20260925 09:16:36.505729 15624 sys_catalog.cc:564] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:16:36.507913 15630 sys_catalog.cc:455] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eef4711ed1144c2a8b200d496d3699b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef4711ed1144c2a8b200d496d3699b5" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46341 } } }
I20260925 09:16:36.508893 15630 sys_catalog.cc:458] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:36.508355 15631 sys_catalog.cc:455] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eef4711ed1144c2a8b200d496d3699b5. Latest consensus state: current_term: 1 leader_uuid: "eef4711ed1144c2a8b200d496d3699b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef4711ed1144c2a8b200d496d3699b5" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46341 } } }
I20260925 09:16:36.509444 15631 sys_catalog.cc:458] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:36.510911 15637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:16:36.516686 15637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:16:36.517367 15624 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 09:16:36.534526 15637 catalog_manager.cc:1347] Generated new cluster ID: 0776da714ec44ce08cd46c2c232d1243
I20260925 09:16:36.534668 15637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:16:36.564903 15637 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:16:36.566387 15637 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:16:36.581348 15637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5: Generated new TSK 0
I20260925 09:16:36.582410 15637 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:16:37.321830 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.19.84:52457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46341
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 09:16:37 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790327797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 09:16:37.751274 15670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:37.751745 15670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:37.751915 15670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:37.763501 15670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:37.763648 15670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:37.763839 15670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:16:37.779522 15670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.19.84:52457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:37.781397 15670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:37.783188 15670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:37.804159 15680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:37.805331 15681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:37.813050 15683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:37.813473 15670 server_base.cc:1034] running on GCE node
I20260925 09:16:37.814491 15670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:37.816247 15670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:37.817533 15670 hybrid_clock.cc:648] HybridClock initialized: now 1790327797817464 us; error 60 us; skew 500 ppm
I20260925 09:16:37.817909 15670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:37 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:37.821249 15670 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:16:37.823041 15670 webserver.cc:460] Webserver started at http://127.3.19.65:42139/ using document root <none> and password file <none>
I20260925 09:16:37.823760 15670 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:37.823896 15670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:37.824213 15670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 09:16:37 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:16:37.842571 15670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "2851853ad1a84031b9010eb715194fca"
format_stamp: "Formatted at 2026-09-25 09:16:37 on dist-test-slave-l6qh"
server_key: "316c091db5b7f087280e7bdbcff24837"
server_key_iv: "bd1cd296fef09f6e4b0090c72cd7b809"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:37.843711 15670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "2851853ad1a84031b9010eb715194fca"
format_stamp: "Formatted at 2026-09-25 09:16:37 on dist-test-slave-l6qh"
server_key: "316c091db5b7f087280e7bdbcff24837"
server_key_iv: "bd1cd296fef09f6e4b0090c72cd7b809"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:37.856868 15670 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.006s sys 0.008s
I20260925 09:16:37.895965 15690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:37.897974 15670 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.009s sys 0.008s
I20260925 09:16:37.898247 15670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "2851853ad1a84031b9010eb715194fca"
format_stamp: "Formatted at 2026-09-25 09:16:37 on dist-test-slave-l6qh"
server_key: "316c091db5b7f087280e7bdbcff24837"
server_key_iv: "bd1cd296fef09f6e4b0090c72cd7b809"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:37.898629 15670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:37.950059 15670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:37.955035 15670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:37.955399 15670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:37.960404 15670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:16:37.961999 15670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:16:37.962105 15670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:37.962250 15670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:16:37.962332 15670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 09:16:38 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:16:38.129607 15802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:34231 every 8 connection(s)
I20260925 09:16:38.129971 15670 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:34231
I20260925 09:16:38.131765 15670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:16:38.136089 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 15670
I20260925 09:16:38.136303 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:16:38.137069 3149 external_mini_cluster.cc:1428] Setting key 1b4623379f9ddaad022451f1e5d8621d
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
Sep 25 09:16:38 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:38.203210 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:38.140837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:54673 (local address 127.3.19.126:46341)
0925 09:16:38.169908 (+ 29071us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:38.169915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:38.175470 (+ 5555us) server_negotiation.cc:407] Connection header received
0925 09:16:38.175556 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:38.175561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:38.175682 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:38.175801 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:38.178875 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:38.180069 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:38.187131 (+ 7062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:38.187706 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:38.187837 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:38.194451 (+ 6614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:38.194460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:38.194465 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:38.194526 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:38.198852 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:38.200104 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:38.200112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:38.200117 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:38.200327 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:38.200966 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:38.200973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:38.200977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:38.201413 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:38.201533 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:38.202571 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0925 09:16:38.202790 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28904,"thread_start_us":29071,"threads_started":1}
I20260925 09:16:38.205950 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:38.140269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46341 (local address 127.3.19.65:54673)
0925 09:16:38.171448 (+ 31179us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:38.171509 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:38.174041 (+ 2532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:38.176997 (+ 2956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:38.177011 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:38.177704 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:38.178614 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:38.178659 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:38.180322 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:38.180327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:38.182566 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:38.182580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:38.188218 (+ 5638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:38.188225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:38.188534 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:38.189508 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:38.189539 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:38.194102 (+ 4563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:38.199112 (+ 5010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:38.199123 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:38.199148 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:38.199939 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:38.200504 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:38.200509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:38.200514 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:38.200817 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:38.201633 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:38.201641 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:38.202092 (+ 451us) client_negotiation.cc:769] Sending connection context
0925 09:16:38.203830 (+ 1738us) client_negotiation.cc:240] Negotiation successful
0925 09:16:38.204405 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30807,"thread_start_us":34144,"threads_started":1}
I20260925 09:16:38.207990 15803 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46341
I20260925 09:16:38.208395 15803 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:38.209376 15803 heartbeater.cc:511] Master 127.3.19.126:46341 requested a full tablet report, sending...
I20260925 09:16:38.212039 15589 ts_manager.cc:194] Registered new tserver with Master: 2851853ad1a84031b9010eb715194fca (127.3.19.65:34231)
I20260925 09:16:38.214171 15589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:54673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:16:38.232275 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.19.84:52457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46341
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 09:16:38 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
W20260925 09:16:38.672202 15809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:38.672514 15809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:38.672674 15809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:38.680222 15809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:38.680372 15809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:38.680543 15809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:16:38.690523 15809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.19.84:52457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:38.691946 15809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:38.693198 15809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:38.712507 15823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:38.712581 15826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:38.712621 15824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:38.714484 15809 server_base.cc:1034] running on GCE node
I20260925 09:16:38.715139 15809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:38.716178 15809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:38.717371 15809 hybrid_clock.cc:648] HybridClock initialized: now 1790327798717322 us; error 46 us; skew 500 ppm
I20260925 09:16:38.717725 15809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:38 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327798, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:38.721119 15809 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:16:38.722744 15809 webserver.cc:460] Webserver started at http://127.3.19.66:37751/ using document root <none> and password file <none>
I20260925 09:16:38.723434 15809 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:38.723556 15809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:38.723853 15809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 09:16:38 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327798, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:16:38.738659 15809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "54fca7eb395b409a99e7018ebe49a1fd"
format_stamp: "Formatted at 2026-09-25 09:16:38 on dist-test-slave-l6qh"
server_key: "923d9c5474db5b0a24d47d915a8c625b"
server_key_iv: "3ea2f4e0ee07ac738bb29dab6aa00da1"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:38.739485 15809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "54fca7eb395b409a99e7018ebe49a1fd"
format_stamp: "Formatted at 2026-09-25 09:16:38 on dist-test-slave-l6qh"
server_key: "923d9c5474db5b0a24d47d915a8c625b"
server_key_iv: "3ea2f4e0ee07ac738bb29dab6aa00da1"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:38.744803 15809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:16:38.764894 15833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:38.766773 15809 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.003s
I20260925 09:16:38.766987 15809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "54fca7eb395b409a99e7018ebe49a1fd"
format_stamp: "Formatted at 2026-09-25 09:16:38 on dist-test-slave-l6qh"
server_key: "923d9c5474db5b0a24d47d915a8c625b"
server_key_iv: "3ea2f4e0ee07ac738bb29dab6aa00da1"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:38.767279 15809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:38.796607 15809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:38.800572 15809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:38.800872 15809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:38.801954 15809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:16:38.803403 15809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:16:38.803501 15809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:38.803651 15809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:16:38.803721 15809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:38.838393 15809 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:46539
I20260925 09:16:38.838397 15945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:46539 every 8 connection(s)
I20260925 09:16:38.839766 15809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:16:38.847792 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 15809
I20260925 09:16:38.847980 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:16:38.848526 3149 external_mini_cluster.cc:1428] Setting key b817b67e5ef171200efe57bb70a64871
Sep 25 09:16:38 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327798, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:38.859581 15948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:38.842536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:49289 (local address 127.3.19.126:46341)
0925 09:16:38.843041 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:38.843045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:38.844091 (+ 1046us) server_negotiation.cc:407] Connection header received
0925 09:16:38.845998 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:38.846003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:38.846102 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:38.846214 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:38.847568 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:38.848432 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:38.850330 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:38.850732 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:38.850878 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:38.854054 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:38.854059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:38.854062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:38.854091 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:38.856659 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:38.857365 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:38.857369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:38.857371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:38.857483 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:38.857910 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:38.857915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:38.857919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:38.858178 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:38.858328 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:38.858966 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 09:16:38.859187 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":287,"threads_started":1}
I20260925 09:16:38.860785 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:38.843157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46341 (local address 127.3.19.66:49289)
0925 09:16:38.843934 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:38.843975 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:38.845735 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:38.846451 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:38.846459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:38.846855 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:38.847404 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:38.847420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:38.848578 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:38.848588 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:38.850188 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:38.850202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:38.850953 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:38.850960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:38.851180 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:38.851828 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:38.851850 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:38.853852 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:38.856795 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:38.856803 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:38.856820 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:38.857211 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:38.857586 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:38.857592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:38.857599 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:38.857796 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:38.858307 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:38.858315 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:38.858611 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 09:16:38.858862 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 09:16:38.859365 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":315,"threads_started":1}
I20260925 09:16:38.862625 15946 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46341
I20260925 09:16:38.862942 15946 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:38.863785 15946 heartbeater.cc:511] Master 127.3.19.126:46341 requested a full tablet report, sending...
I20260925 09:16:38.865413 15589 ts_manager.cc:194] Registered new tserver with Master: 54fca7eb395b409a99e7018ebe49a1fd (127.3.19.66:46539)
I20260925 09:16:38.866453 15589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:49289
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:16:38.908598 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.19.84:52457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46341
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:16:39.217963 15803 heartbeater.cc:503] Master 127.3.19.126:46341 was elected leader, sending a full tablet report...
W20260925 09:16:39.233579 15952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:39.233852 15952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:39.234028 15952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:39.241298 15952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:39.241422 15952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:39.241603 15952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:16:39.250929 15952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.19.84:52457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:39.252156 15952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:39.253430 15952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:39.266160 15960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:39.266211 15957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:39.266188 15958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:39.267086 15952 server_base.cc:1034] running on GCE node
I20260925 09:16:39.267630 15952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:39.268471 15952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:39.269656 15952 hybrid_clock.cc:648] HybridClock initialized: now 1790327799269605 us; error 48 us; skew 500 ppm
I20260925 09:16:39.269924 15952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:39 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:39.272953 15952 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:16:39.274374 15952 webserver.cc:460] Webserver started at http://127.3.19.67:42879/ using document root <none> and password file <none>
I20260925 09:16:39.275004 15952 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:39.275134 15952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:39.275441 15952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 09:16:39 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:16:39.289604 15952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "9c78be255e794292b8e2d7a74cf73672"
format_stamp: "Formatted at 2026-09-25 09:16:39 on dist-test-slave-l6qh"
server_key: "d6c77101ba5b4bea937da447575d45a9"
server_key_iv: "540c5c91210852ba92706da06fd1dd53"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:39.290437 15952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "9c78be255e794292b8e2d7a74cf73672"
format_stamp: "Formatted at 2026-09-25 09:16:39 on dist-test-slave-l6qh"
server_key: "d6c77101ba5b4bea937da447575d45a9"
server_key_iv: "540c5c91210852ba92706da06fd1dd53"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:39.296032 15952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:16:39.313072 15967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:39.314682 15952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 09:16:39.314862 15952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "9c78be255e794292b8e2d7a74cf73672"
format_stamp: "Formatted at 2026-09-25 09:16:39 on dist-test-slave-l6qh"
server_key: "d6c77101ba5b4bea937da447575d45a9"
server_key_iv: "540c5c91210852ba92706da06fd1dd53"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:39.315062 15952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:39.339869 15952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:39.343742 15952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:39.344043 15952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:39.344980 15952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:16:39.346287 15952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:16:39.346357 15952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:39.346519 15952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:16:39.346571 15952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:39.377868 15952 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:34735
I20260925 09:16:39.377910 16079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:34735 every 8 connection(s)
I20260925 09:16:39.379101 15952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:16:39.384565 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 15952
I20260925 09:16:39.384753 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:16:39.385222 3149 external_mini_cluster.cc:1428] Setting key fced5b2b907161c0b9578e6d7d776f83
Sep 25 09:16:39 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:39.398511 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:39.381540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:59193 (local address 127.3.19.126:46341)
0925 09:16:39.381993 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:39.381999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:39.382998 (+ 999us) server_negotiation.cc:407] Connection header received
0925 09:16:39.384610 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:39.384615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:39.384715 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:39.384842 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:39.386723 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:39.387700 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:39.389525 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:39.389925 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:39.390019 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:39.393152 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:39.393156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:39.393160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:39.393190 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:39.395899 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:39.396582 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:39.396586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:39.396588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:39.396701 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:39.397070 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:39.397073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:39.397074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:39.397399 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:39.397509 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:39.398105 (+ 596us) server_negotiation.cc:299] Negotiation successful
0925 09:16:39.398268 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":199,"threads_started":1}
I20260925 09:16:39.399718 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:39.382146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46341 (local address 127.3.19.67:59193)
0925 09:16:39.382845 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:39.382881 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:39.384356 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:39.385066 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:39.385074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:39.385703 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:39.386525 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:39.386552 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:39.387888 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:39.387894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:39.389391 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:39.389403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:39.390114 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:39.390119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:39.390309 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:39.390977 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:39.391001 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:39.392973 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:39.396026 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:39.396032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:39.396045 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:39.396488 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:39.396791 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:39.396794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:39.396797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:39.396983 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:39.397504 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:39.397509 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:39.397809 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 09:16:39.398037 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 09:16:39.398404 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":292,"threads_started":1}
I20260925 09:16:39.401366 16080 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46341
I20260925 09:16:39.401722 16080 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:39.402446 16080 heartbeater.cc:511] Master 127.3.19.126:46341 requested a full tablet report, sending...
I20260925 09:16:39.403962 15588 ts_manager.cc:194] Registered new tserver with Master: 9c78be255e794292b8e2d7a74cf73672 (127.3.19.67:34735)
I20260925 09:16:39.405046 15588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:59193
I20260925 09:16:39.412901 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:16:39.414572 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 15670
Sep 25 09:16:39 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:16:39.430310 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:39.415453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.3.19.65:34231)
0925 09:16:39.416042 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:39.416048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:39.416072 (+ 24us) server_negotiation.cc:407] Connection header received
0925 09:16:39.416124 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:39.416130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:39.416314 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:39.416431 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:39.418219 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:39.419202 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:39.420835 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:39.421416 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:39.421503 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:39.425324 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:39.425334 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:39.425345 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:39.425380 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:39.428075 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:39.428663 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:39.428668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:39.428670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:39.428799 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:39.429093 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:39.429096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:39.429097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:39.429463 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:39.429673 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:39.429858 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 09:16:39.430044 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":301,"threads_started":1}
I20260925 09:16:39.431398 15798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:39.513370 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 15670
I20260925 09:16:39.529002 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 15809
Sep 25 09:16:39 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:16:39.545112 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:39.529902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.3.19.66:46539)
0925 09:16:39.530456 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:39.530463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:39.530488 (+ 25us) server_negotiation.cc:407] Connection header received
0925 09:16:39.530548 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:39.530554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:39.530766 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:39.530890 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:39.532020 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:39.532990 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:39.534274 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:39.534814 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:39.534914 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:39.539060 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:39.539070 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:39.539084 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:39.539126 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:39.542444 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:39.543104 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:39.543113 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:39.543115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:39.543282 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:39.543632 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:39.543638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:39.543641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:39.544005 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:39.544278 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:39.544468 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 09:16:39.544717 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":294,"threads_started":1}
I20260925 09:16:39.546425 15941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:39.628839 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 15809
I20260925 09:16:39.645315 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 15952
Sep 25 09:16:39 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:16:39.665184 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:39.646445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.3.19.67:34735)
0925 09:16:39.647131 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:39.647142 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:39.647175 (+ 33us) server_negotiation.cc:407] Connection header received
0925 09:16:39.647247 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:39.647254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:39.647471 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:39.647612 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:39.649003 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:39.650949 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:39.653189 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:39.654112 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:39.654249 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:39.659220 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:39.659231 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:39.659245 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:39.659280 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:39.662305 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:39.662979 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:39.662990 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:39.662999 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:39.663121 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:39.663515 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:39.663520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:39.663523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:39.663865 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:39.664183 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:39.664543 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 09:16:39.664818 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":369,"threads_started":1}
I20260925 09:16:39.666483 16075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:39.749647 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 15952
I20260925 09:16:39.765358 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 15557
I20260925 09:16:39.766737 15619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:39.873407 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 15557
I20260925 09:16:39.894400 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46341
--webserver_interface=127.3.19.126
--webserver_port=46163
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:46341
--ranger_config_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.19.84:52457/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.19.84:52457/kms with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:16:40.208587 16092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:40.208894 16092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:40.209048 16092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:40.215875 16092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:16:40.216012 16092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:40.216113 16092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:40.216212 16092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:16:40.216308 16092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:16:40.225631 16092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.19.84:52457/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:46341
--ranger_config_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=46163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:40.226794 16092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:40.228032 16092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:40.239638 16097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:40.239598 16098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:40.239562 16100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:40.240376 16092 server_base.cc:1034] running on GCE node
I20260925 09:16:40.241001 16092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:40.242336 16092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:40.243530 16092 hybrid_clock.cc:648] HybridClock initialized: now 1790327800243490 us; error 33 us; skew 500 ppm
I20260925 09:16:40.243827 16092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:40 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:40.246703 16092 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:16:40.248065 16092 webserver.cc:460] Webserver started at http://127.3.19.126:46163/ using document root <none> and password file <none>
I20260925 09:16:40.248672 16092 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:40.248823 16092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 09:16:40 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:16:40.297492 16092 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 09:16:40.300588 16108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:40.302052 16092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:16:40.302237 16092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "eef4711ed1144c2a8b200d496d3699b5"
format_stamp: "Formatted at 2026-09-25 09:16:36 on dist-test-slave-l6qh"
server_key: "cbb49c42b4a773e2474e8bdfc7238e8e"
server_key_iv: "3a3602aa82230c70870ce0d335f72775"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:40.303155 16092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:40.321861 16092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:40.325866 16092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:40.326143 16092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:40.345602 16092 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:46341
I20260925 09:16:40.345623 16160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:46341 every 8 connection(s)
I20260925 09:16:40.347000 16092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:16:40.353782 16161 sys_catalog.cc:263] Verifying existing consensus state
I20260925 09:16:40.355643 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 16092
I20260925 09:16:40.356678 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.19.84:52457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:34231
--local_ip_for_outbound_sockets=127.3.19.65
--tserver_master_addrs=127.3.19.126:46341
--webserver_port=42139
--webserver_interface=127.3.19.65
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.19.84:52457/kms with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:16:40.357002 16161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5: Bootstrap starting.
I20260925 09:16:40.379768 16161 log.cc:824] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:40.391777 16161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 09:16:40.392390 16161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5: Bootstrap complete.
I20260925 09:16:40.405581 16161 raft_consensus.cc:348] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef4711ed1144c2a8b200d496d3699b5" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46341 } }
I20260925 09:16:40.406813 16161 raft_consensus.cc:729] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef4711ed1144c2a8b200d496d3699b5, State: Initialized, Role: FOLLOWER
I20260925 09:16:40.407533 16161 consensus_queue.cc:260] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef4711ed1144c2a8b200d496d3699b5" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46341 } }
I20260925 09:16:40.407794 16161 raft_consensus.cc:388] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:16:40.407905 16161 raft_consensus.cc:482] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:16:40.408083 16161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 09:16:40.413357 16161 raft_consensus.cc:504] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef4711ed1144c2a8b200d496d3699b5" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46341 } }
I20260925 09:16:40.413909 16161 leader_election.cc:302] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eef4711ed1144c2a8b200d496d3699b5; no voters:
I20260925 09:16:40.414568 16161 leader_election.cc:288] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 09:16:40.414803 16165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 09:16:40.422916 16161 sys_catalog.cc:564] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:16:40.425536 16165 raft_consensus.cc:686] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [term 2 LEADER]: Becoming Leader. State: Replica: eef4711ed1144c2a8b200d496d3699b5, State: Running, Role: LEADER
I20260925 09:16:40.426120 16165 consensus_queue.cc:237] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef4711ed1144c2a8b200d496d3699b5" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46341 } }
I20260925 09:16:40.436085 16166 sys_catalog.cc:455] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eef4711ed1144c2a8b200d496d3699b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef4711ed1144c2a8b200d496d3699b5" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46341 } } }
I20260925 09:16:40.436110 16167 sys_catalog.cc:455] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eef4711ed1144c2a8b200d496d3699b5. Latest consensus state: current_term: 2 leader_uuid: "eef4711ed1144c2a8b200d496d3699b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef4711ed1144c2a8b200d496d3699b5" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46341 } } }
I20260925 09:16:40.436532 16166 sys_catalog.cc:458] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:40.436533 16167 sys_catalog.cc:458] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:40.451354 16173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:16:40.464964 16173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:16:40.466360 16173 catalog_manager.cc:1259] Loaded cluster ID: 0776da714ec44ce08cd46c2c232d1243
I20260925 09:16:40.466527 16173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:16:40.470450 16173 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:16:40.471845 16173 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5: Loaded TSK: 0
I20260925 09:16:40.473318 16173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:16:40.490332 16161 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260925 09:16:40.849328 16163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:40.849771 16163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:40.849941 16163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:40.862990 16163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:40.863152 16163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:40.863339 16163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:16:40.879158 16163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.19.84:52457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:34231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=42139
--tserver_master_addrs=127.3.19.126:46341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:40.880875 16163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:40.882663 16163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:40.914350 16199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:40.915295 16200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:40.937536 16163 server_base.cc:1034] running on GCE node
W20260925 09:16:40.938561 16202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:40.939476 16163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:40.950997 16163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:40.952746 16163 hybrid_clock.cc:648] HybridClock initialized: now 1790327800952661 us; error 92 us; skew 500 ppm
I20260925 09:16:40.953164 16163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:40 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:40.957613 16163 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:16:40.969630 16163 webserver.cc:460] Webserver started at http://127.3.19.65:42139/ using document root <none> and password file <none>
I20260925 09:16:40.970690 16163 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:40.970860 16163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 09:16:40 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:16:41.003602 16163 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.004s
I20260925 09:16:41.009956 16210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:41.011991 16163 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.001s
I20260925 09:16:41.012210 16163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "2851853ad1a84031b9010eb715194fca"
format_stamp: "Formatted at 2026-09-25 09:16:37 on dist-test-slave-l6qh"
server_key: "316c091db5b7f087280e7bdbcff24837"
server_key_iv: "bd1cd296fef09f6e4b0090c72cd7b809"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:41.013676 16163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:41.073920 16163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:41.079344 16163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:41.079676 16163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:41.080932 16163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:16:41.082686 16163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:16:41.082815 16163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:41.082924 16163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:16:41.082996 16163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:41.148257 16163 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:34231
I20260925 09:16:41.150095 16163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:16:41.153365 16324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:34231 every 8 connection(s)
I20260925 09:16:41.156543 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 16163
I20260925 09:16:41.157380 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.19.84:52457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:46539
--local_ip_for_outbound_sockets=127.3.19.66
--tserver_master_addrs=127.3.19.126:46341
--webserver_port=37751
--webserver_interface=127.3.19.66
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.19.84:52457/kms with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 09:16:41 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:41.196767 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:41.159737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46341 (local address 127.3.19.65:43727)
0925 09:16:41.166413 (+ 6676us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:41.166465 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:41.172026 (+ 5561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:41.174457 (+ 2431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:41.174471 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:41.175148 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:41.175952 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:41.175979 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:41.178481 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:41.178489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:41.180702 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:41.180725 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:41.182509 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:41.182516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:41.182771 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:41.183554 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:41.183582 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:41.186481 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:41.191240 (+ 4759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:41.191248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:41.191265 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:41.191869 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:41.192441 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:41.192446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:41.192449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:41.192713 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:41.193885 (+ 1172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:41.193893 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:41.194325 (+ 432us) client_negotiation.cc:769] Sending connection context
0925 09:16:41.194676 (+ 351us) client_negotiation.cc:240] Negotiation successful
0925 09:16:41.195300 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6395,"thread_start_us":6144,"threads_started":1}
I20260925 09:16:41.199175 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:41.161232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:43727 (local address 127.3.19.126:46341)
0925 09:16:41.165802 (+ 4570us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:41.165828 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:41.171506 (+ 5678us) server_negotiation.cc:407] Connection header received
0925 09:16:41.172701 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:41.172725 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:41.173217 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:41.173660 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:41.176183 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:41.178282 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:41.180875 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:41.181916 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:41.182101 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:41.186779 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:41.186789 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:41.186810 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:41.186857 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:41.191037 (+ 4180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:41.192046 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:41.192056 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:41.192065 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:41.192289 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:41.192858 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:41.192864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:41.192867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:41.193583 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:41.193788 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:41.197524 (+ 3736us) server_negotiation.cc:299] Negotiation successful
0925 09:16:41.197954 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4221,"thread_start_us":3967,"threads_started":1}
I20260925 09:16:41.203105 16325 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46341
I20260925 09:16:41.203415 16325 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:41.204069 16325 heartbeater.cc:511] Master 127.3.19.126:46341 requested a full tablet report, sending...
W20260925 09:16:41.207021 16325 heartbeater.cc:644] Failed to heartbeat to 127.3.19.126:46341 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 09:16:41.207296 16325 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:41.207615 16325 heartbeater.cc:511] Master 127.3.19.126:46341 requested a full tablet report, sending...
I20260925 09:16:41.208752 16325 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:41.209029 16325 heartbeater.cc:511] Master 127.3.19.126:46341 requested a full tablet report, sending...
W20260925 09:16:41.210186 16325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 09:16:41.576150 16329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:41.576452 16329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:41.576627 16329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:41.583930 16329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:41.584084 16329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:41.584306 16329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:16:41.594240 16329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.19.84:52457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:46539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=37751
--tserver_master_addrs=127.3.19.126:46341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:41.595566 16329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:41.596859 16329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:41.615047 16342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:41.615809 16341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:41.620703 16344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:41.621603 16329 server_base.cc:1034] running on GCE node
I20260925 09:16:41.622342 16329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:41.623577 16329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:41.624840 16329 hybrid_clock.cc:648] HybridClock initialized: now 1790327801624751 us; error 72 us; skew 500 ppm
I20260925 09:16:41.625267 16329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:41 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327801, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:41.635502 16329 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:16:41.637465 16329 webserver.cc:460] Webserver started at http://127.3.19.66:37751/ using document root <none> and password file <none>
I20260925 09:16:41.638413 16329 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:41.638579 16329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 09:16:41 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327801, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:16:41.668743 16329 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.004s
I20260925 09:16:41.673380 16354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:41.675251 16329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260925 09:16:41.675477 16329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "54fca7eb395b409a99e7018ebe49a1fd"
format_stamp: "Formatted at 2026-09-25 09:16:38 on dist-test-slave-l6qh"
server_key: "923d9c5474db5b0a24d47d915a8c625b"
server_key_iv: "3ea2f4e0ee07ac738bb29dab6aa00da1"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:41.676569 16329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:41.697716 16329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:41.703274 16329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:41.703625 16329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:41.704885 16329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:16:41.706836 16329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:16:41.707072 16329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:41.707257 16329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:16:41.707377 16329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:41.803215 16329 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:46539
I20260925 09:16:41.803310 16470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:46539 every 8 connection(s)
I20260925 09:16:41.806033 16329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:16:41.813522 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 16329
I20260925 09:16:41.814370 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.19.84:52457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:34735
--local_ip_for_outbound_sockets=127.3.19.67
--tserver_master_addrs=127.3.19.126:46341
--webserver_port=42879
--webserver_interface=127.3.19.67
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.19.84:52457/kms with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 09:16:41 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327801, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:41.837324 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:41.810659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:35007 (local address 127.3.19.126:46341)
0925 09:16:41.815986 (+ 5327us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:41.815992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:41.816012 (+ 20us) server_negotiation.cc:407] Connection header received
0925 09:16:41.816075 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:41.816080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:41.816156 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:41.816259 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:41.820325 (+ 4066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:41.821582 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:41.824254 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:41.824933 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:41.825034 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:41.829437 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:41.829444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:41.829449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:41.829491 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:41.833365 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:41.834266 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:41.834272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:41.834276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:41.834419 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:41.834953 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:41.834959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:41.834962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:41.835302 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:41.835439 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:41.836393 (+ 954us) server_negotiation.cc:299] Negotiation successful
0925 09:16:41.836612 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5197,"thread_start_us":6816,"threads_started":1}
I20260925 09:16:41.839581 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:41.810028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46341 (local address 127.3.19.66:35007)
0925 09:16:41.812403 (+ 2375us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:41.812451 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:41.814852 (+ 2401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:41.818380 (+ 3528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:41.818392 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:41.819204 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:41.820114 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:41.820139 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:41.821803 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:41.821808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:41.824098 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:41.824112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:41.825393 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:41.825400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:41.825669 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:41.826448 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:41.826477 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:41.829110 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:41.833524 (+ 4414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:41.833533 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:41.833551 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:41.834129 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:41.834558 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:41.834563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:41.834567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:41.834824 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:41.835537 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:41.835544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:41.835957 (+ 413us) client_negotiation.cc:769] Sending connection context
0925 09:16:41.838082 (+ 2125us) client_negotiation.cc:240] Negotiation successful
0925 09:16:41.838552 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2071,"thread_start_us":1862,"threads_started":1}
I20260925 09:16:41.841377 16471 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46341
I20260925 09:16:41.841729 16471 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:41.842680 16471 heartbeater.cc:511] Master 127.3.19.126:46341 requested a full tablet report, sending...
I20260925 09:16:41.845002 16125 ts_manager.cc:194] Registered new tserver with Master: 54fca7eb395b409a99e7018ebe49a1fd (127.3.19.66:46539)
I20260925 09:16:41.847411 16125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:35007
Sep 25 09:16:41 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790327801, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:42.210331 16325 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:42.210794 16325 heartbeater.cc:511] Master 127.3.19.126:46341 requested a full tablet report, sending...
I20260925 09:16:42.212080 16125 ts_manager.cc:194] Registered new tserver with Master: 2851853ad1a84031b9010eb715194fca (127.3.19.65:34231)
I20260925 09:16:42.213299 16125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:43727
Sep 25 09:16:42 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327801, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
W20260925 09:16:42.291196 16475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:42.291585 16475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:42.291738 16475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:42.302407 16475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:42.302556 16475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:42.302740 16475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:16:42.313889 16475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.19.84:52457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:34735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=42879
--tserver_master_addrs=127.3.19.126:46341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:42.315364 16475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:42.316766 16475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:42.334631 16481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:42.335500 16482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:42.343230 16485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:42.344059 16475 server_base.cc:1034] running on GCE node
I20260925 09:16:42.344746 16475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:42.355386 16475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:42.358758 16475 hybrid_clock.cc:648] HybridClock initialized: now 1790327802358683 us; error 73 us; skew 500 ppm
I20260925 09:16:42.359104 16475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:42 dist-test-slave-l6qh krb5kdc[15109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:42.362411 16475 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:16:42.369606 16475 webserver.cc:460] Webserver started at http://127.3.19.67:42879/ using document root <none> and password file <none>
I20260925 09:16:42.370539 16475 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:42.370689 16475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 09:16:42 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
I20260925 09:16:42.400238 16475 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 09:16:42.403712 16493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:42.405536 16475 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 09:16:42.405707 16475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "9c78be255e794292b8e2d7a74cf73672"
format_stamp: "Formatted at 2026-09-25 09:16:39 on dist-test-slave-l6qh"
server_key: "d6c77101ba5b4bea937da447575d45a9"
server_key_iv: "540c5c91210852ba92706da06fd1dd53"
server_key_version: "kuduclusterkey@0"
I20260925 09:16:42.407109 16475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:42.442454 16475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:42.447456 16475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:42.447757 16475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:42.448935 16475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:16:42.450625 16475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:16:42.450749 16475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:42.450856 16475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:16:42.450933 16475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:42.515550 16475 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:34735
I20260925 09:16:42.517053 16605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:34735 every 8 connection(s)
I20260925 09:16:42.517509 16475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:16:42.525833 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 16475
Sep 25 09:16:42 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327801, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for HTTP/127.3.19.84@KRBTEST.COM
Sep 25 09:16:42 dist-test-slave-l6qh krb5kdc[15109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:42.581131 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.526898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46341 (local address 127.3.19.67:55377)
0925 09:16:42.551674 (+ 24776us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:42.551758 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:42.555232 (+ 3474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:42.556677 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:42.556691 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:42.557379 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:42.558248 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:42.558272 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.559694 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.559701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:42.562354 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:42.562368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.563564 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.563572 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:42.563860 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.565881 (+ 2021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:42.565920 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:42.569128 (+ 3208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:42.576559 (+ 7431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:42.576568 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:42.576586 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:42.577134 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:42.577617 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:42.577621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:42.577624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:42.577859 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:42.578543 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:42.578551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:42.578964 (+ 413us) client_negotiation.cc:769] Sending connection context
0925 09:16:42.579258 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 09:16:42.579810 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24344,"thread_start_us":24081,"threads_started":1}
I20260925 09:16:42.582185 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.527298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:55377 (local address 127.3.19.126:46341)
0925 09:16:42.548900 (+ 21602us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:42.548910 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:42.554271 (+ 5361us) server_negotiation.cc:407] Connection header received
0925 09:16:42.555862 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:42.555871 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:42.556008 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:42.556133 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:42.558471 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.559521 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.562501 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.563125 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.563255 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.569420 (+ 6165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:42.569427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:42.569432 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:42.569470 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:42.576391 (+ 6921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:42.577305 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:42.577313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:42.577316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:42.577448 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:42.577980 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:42.577984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:42.577987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:42.578353 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:42.578479 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:42.581616 (+ 3137us) server_negotiation.cc:299] Negotiation successful
0925 09:16:42.581835 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21414,"thread_start_us":22705,"threads_started":1}
I20260925 09:16:42.582683 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.527608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.3.19.126:46341)
0925 09:16:42.550301 (+ 22693us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:42.550310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:42.556572 (+ 6262us) server_negotiation.cc:407] Connection header received
0925 09:16:42.556735 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:42.556740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:42.556824 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:42.556931 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:42.560997 (+ 4066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.562196 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.564128 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.564799 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.564922 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.574836 (+ 9914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:42.574855 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:42.574859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:42.574912 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:42.578560 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:42.579499 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:42.579504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:42.579507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:42.579649 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:42.580088 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:42.580092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:42.580095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:42.580436 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:42.580699 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:42.581718 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0925 09:16:42.582258 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":273,"server-negotiator.queue_time_us":22536,"thread_start_us":20994,"threads_started":1}
I20260925 09:16:42.584033 16606 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46341
I20260925 09:16:42.584334 16606 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:42.585101 16606 heartbeater.cc:511] Master 127.3.19.126:46341 requested a full tablet report, sending...
I20260925 09:16:42.586691 16124 ts_manager.cc:194] Registered new tserver with Master: 9c78be255e794292b8e2d7a74cf73672 (127.3.19.67:34735)
I20260925 09:16:42.587859 16124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:55377
I20260925 09:16:42.598469 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:16:42.656702 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.626974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.3.19.126:46341)
0925 09:16:42.627257 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:42.627267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:42.639637 (+ 12370us) server_negotiation.cc:407] Connection header received
0925 09:16:42.639849 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:42.639855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:42.639982 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:42.640123 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:42.642800 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.644023 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.646212 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.646921 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.647058 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.650715 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:42.650724 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:42.650729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:42.650773 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:42.653751 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:42.654640 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:42.654645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:42.654649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:42.654841 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:42.655231 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:42.655236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:42.655240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:42.655530 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:42.655796 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:42.656125 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 09:16:42.656349 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 09:16:42.662345 16124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:16:42.672557 16124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:16:42.708920 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.691317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:46539 (local address 127.0.0.1:45634)
0925 09:16:42.698290 (+ 6973us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:42.698340 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:42.698539 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:42.699858 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:42.699869 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:42.699911 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:42.700514 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:42.700532 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.703066 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.703075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:42.705851 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:42.705865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.707966 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.707973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:42.708166 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.708222 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 09:16:42.708357 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 09:16:42.708554 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6693,"thread_start_us":7349,"threads_started":1}
I20260925 09:16:42.709936 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.691871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45634 (local address 127.3.19.66:46539)
0925 09:16:42.697687 (+ 5816us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:42.697698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:42.698801 (+ 1103us) server_negotiation.cc:407] Connection header received
0925 09:16:42.698909 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:42.698919 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:42.699250 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:42.699502 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:42.700761 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.702840 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.706032 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.707464 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.707588 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.707800 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:42.709460 (+ 1660us) server_negotiation.cc:299] Negotiation successful
0925 09:16:42.709634 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5618,"thread_start_us":5484,"threads_started":1}
I20260925 09:16:42.715385 16404 tablet_service.cc:1467] Processing CreateTablet for tablet deedc4dc6eee468e891184587d592f82 (DEFAULT_TABLE table=test-table [id=50aaa3e150484f98afc7014a8553b13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:16:42.716888 16404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deedc4dc6eee468e891184587d592f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:42.720534 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.702019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:34735 (local address 127.0.0.1:35836)
0925 09:16:42.710485 (+ 8466us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:42.710507 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:42.710611 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:42.712024 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:42.712030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:42.712056 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:42.712327 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:42.712337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.714850 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.714859 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:42.717165 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:42.717180 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.719601 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.719610 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:42.719886 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.719940 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 09:16:42.720039 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 09:16:42.720188 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8355,"thread_start_us":8211,"threads_started":1}
I20260925 09:16:42.721481 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.702641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.3.19.67:34735)
0925 09:16:42.711133 (+ 8492us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:42.711141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:42.711176 (+ 35us) server_negotiation.cc:407] Connection header received
0925 09:16:42.711272 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:42.711280 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:42.711527 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:42.711730 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:42.712544 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.714597 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.717412 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.719005 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.719137 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.719415 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:42.720940 (+ 1525us) server_negotiation.cc:299] Negotiation successful
0925 09:16:42.721103 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8328,"thread_start_us":8165,"threads_started":1}
I20260925 09:16:42.725317 16541 tablet_service.cc:1467] Processing CreateTablet for tablet deedc4dc6eee468e891184587d592f82 (DEFAULT_TABLE table=test-table [id=50aaa3e150484f98afc7014a8553b13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:16:42.726837 16541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deedc4dc6eee468e891184587d592f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:42.742211 16628 tablet_bootstrap.cc:492] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd: Bootstrap starting.
I20260925 09:16:42.746380 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.698435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34231 (local address 127.0.0.1:35810)
0925 09:16:42.709168 (+ 10733us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:42.709187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:42.709305 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:42.729312 (+ 20007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:42.729317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:42.729349 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:42.729541 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:42.729553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.741362 (+ 11809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.741368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:42.743318 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:42.743339 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.745552 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.745563 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:42.745800 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.745847 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 09:16:42.745928 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 09:16:42.746083 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10639,"thread_start_us":24902,"threads_started":1}
I20260925 09:16:42.747627 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.698769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.3.19.65:34231)
0925 09:16:42.724737 (+ 25968us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:42.724745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:42.724775 (+ 30us) server_negotiation.cc:407] Connection header received
0925 09:16:42.724866 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:42.724874 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:42.725103 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:42.725354 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:42.730423 (+ 5069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.732049 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.743585 (+ 11536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.745186 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.746786 (+ 1600us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.747034 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:42.747110 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 09:16:42.747280 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25571,"thread_start_us":25729,"threads_started":1}
I20260925 09:16:42.751426 16259 tablet_service.cc:1467] Processing CreateTablet for tablet deedc4dc6eee468e891184587d592f82 (DEFAULT_TABLE table=test-table [id=50aaa3e150484f98afc7014a8553b13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:16:42.752875 16259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deedc4dc6eee468e891184587d592f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:42.759052 16629 tablet_bootstrap.cc:492] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672: Bootstrap starting.
I20260925 09:16:42.759886 16628 tablet_bootstrap.cc:654] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:42.762730 16629 tablet_bootstrap.cc:654] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:42.763940 16629 log.cc:824] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:42.768687 16629 tablet_bootstrap.cc:492] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672: No bootstrap required, opened a new log
I20260925 09:16:42.768965 16629 ts_tablet_manager.cc:1397] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 09:16:42.761173 16628 log.cc:824] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:42.778159 16628 tablet_bootstrap.cc:492] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd: No bootstrap required, opened a new log
I20260925 09:16:42.778445 16628 ts_tablet_manager.cc:1397] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.000s
I20260925 09:16:42.779439 16629 raft_consensus.cc:348] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54fca7eb395b409a99e7018ebe49a1fd" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 46539 } } peers { permanent_uuid: "2851853ad1a84031b9010eb715194fca" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34231 } } peers { permanent_uuid: "9c78be255e794292b8e2d7a74cf73672" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34735 } }
I20260925 09:16:42.779881 16629 raft_consensus.cc:374] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:42.779996 16629 raft_consensus.cc:729] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c78be255e794292b8e2d7a74cf73672, State: Initialized, Role: FOLLOWER
I20260925 09:16:42.780736 16629 consensus_queue.cc:260] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54fca7eb395b409a99e7018ebe49a1fd" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 46539 } } peers { permanent_uuid: "2851853ad1a84031b9010eb715194fca" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34231 } } peers { permanent_uuid: "9c78be255e794292b8e2d7a74cf73672" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34735 } }
I20260925 09:16:42.786338 16632 tablet_bootstrap.cc:492] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca: Bootstrap starting.
I20260925 09:16:42.789327 16629 ts_tablet_manager.cc:1428] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672: Time spent starting tablet: real 0.020s user 0.011s sys 0.010s
I20260925 09:16:42.789845 16606 heartbeater.cc:503] Master 127.3.19.126:46341 was elected leader, sending a full tablet report...
I20260925 09:16:42.793339 16632 tablet_bootstrap.cc:654] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:42.794950 16632 log.cc:824] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:42.795739 16628 raft_consensus.cc:348] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54fca7eb395b409a99e7018ebe49a1fd" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 46539 } } peers { permanent_uuid: "2851853ad1a84031b9010eb715194fca" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34231 } } peers { permanent_uuid: "9c78be255e794292b8e2d7a74cf73672" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34735 } }
I20260925 09:16:42.796181 16628 raft_consensus.cc:374] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:42.796267 16628 raft_consensus.cc:729] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54fca7eb395b409a99e7018ebe49a1fd, State: Initialized, Role: FOLLOWER
I20260925 09:16:42.796901 16628 consensus_queue.cc:260] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54fca7eb395b409a99e7018ebe49a1fd" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 46539 } } peers { permanent_uuid: "2851853ad1a84031b9010eb715194fca" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34231 } } peers { permanent_uuid: "9c78be255e794292b8e2d7a74cf73672" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34735 } }
I20260925 09:16:42.813428 16633 raft_consensus.cc:482] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:16:42.813777 16633 raft_consensus.cc:504] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54fca7eb395b409a99e7018ebe49a1fd" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 46539 } } peers { permanent_uuid: "2851853ad1a84031b9010eb715194fca" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34231 } } peers { permanent_uuid: "9c78be255e794292b8e2d7a74cf73672" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34735 } }
I20260925 09:16:42.815717 16633 leader_election.cc:288] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54fca7eb395b409a99e7018ebe49a1fd (127.3.19.66:46539), 2851853ad1a84031b9010eb715194fca (127.3.19.65:34231)
I20260925 09:16:42.824580 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.816221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:46539 (local address 127.3.19.67:55561)
0925 09:16:42.816374 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:42.816396 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:42.816513 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:42.817798 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:42.817803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:42.817831 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:42.818034 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:42.818045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.819706 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.819713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:42.822038 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:42.822056 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.823782 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.823789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:42.824005 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.824046 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 09:16:42.824155 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 09:16:42.824286 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 09:16:42.825258 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.817141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:55561 (local address 127.3.19.66:46539)
0925 09:16:42.817346 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:42.817352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:42.817372 (+ 20us) server_negotiation.cc:407] Connection header received
0925 09:16:42.817436 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:42.817442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:42.817523 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:42.817616 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:42.818207 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.819362 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.822244 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.823439 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.823565 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.823660 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:42.824864 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0925 09:16:42.824997 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 09:16:42.826109 16424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deedc4dc6eee468e891184587d592f82" candidate_uuid: "9c78be255e794292b8e2d7a74cf73672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54fca7eb395b409a99e7018ebe49a1fd" is_pre_election: true
I20260925 09:16:42.826689 16424 raft_consensus.cc:2454] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c78be255e794292b8e2d7a74cf73672 in term 0.
I20260925 09:16:42.827401 16497 leader_election.cc:302] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54fca7eb395b409a99e7018ebe49a1fd, 9c78be255e794292b8e2d7a74cf73672; no voters:
I20260925 09:16:42.827839 16633 raft_consensus.cc:2781] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:16:42.828040 16633 raft_consensus.cc:482] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:16:42.828168 16633 raft_consensus.cc:3037] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:42.831493 16633 raft_consensus.cc:504] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54fca7eb395b409a99e7018ebe49a1fd" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 46539 } } peers { permanent_uuid: "2851853ad1a84031b9010eb715194fca" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34231 } } peers { permanent_uuid: "9c78be255e794292b8e2d7a74cf73672" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34735 } }
I20260925 09:16:42.833992 16424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deedc4dc6eee468e891184587d592f82" candidate_uuid: "9c78be255e794292b8e2d7a74cf73672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54fca7eb395b409a99e7018ebe49a1fd"
I20260925 09:16:42.834208 16424 raft_consensus.cc:3037] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:42.837478 16633 leader_election.cc:288] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [CANDIDATE]: Term 1 election: Requested vote from peers 54fca7eb395b409a99e7018ebe49a1fd (127.3.19.66:46539), 2851853ad1a84031b9010eb715194fca (127.3.19.65:34231)
I20260925 09:16:42.837931 16424 raft_consensus.cc:2454] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c78be255e794292b8e2d7a74cf73672 in term 1.
I20260925 09:16:42.838346 16497 leader_election.cc:302] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54fca7eb395b409a99e7018ebe49a1fd, 9c78be255e794292b8e2d7a74cf73672; no voters:
I20260925 09:16:42.838711 16633 raft_consensus.cc:2781] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:16:42.843006 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.816940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:33901 (local address 127.3.19.65:34231)
0925 09:16:42.817190 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:42.817199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:42.822797 (+ 5598us) server_negotiation.cc:407] Connection header received
0925 09:16:42.823095 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:42.823103 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:42.823207 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:42.823273 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:42.833369 (+ 10096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.834715 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.839111 (+ 4396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.840722 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.840848 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.840988 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:42.842490 (+ 1502us) server_negotiation.cc:299] Negotiation successful
0925 09:16:42.842684 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 09:16:42.843930 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.816726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34231 (local address 127.3.19.67:33901)
0925 09:16:42.822682 (+ 5956us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:42.822712 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:42.822905 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:42.823485 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:42.823492 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:42.823522 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:42.824089 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:42.824106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.834945 (+ 10839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.834953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:42.837546 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:42.837562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.841323 (+ 3761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.841332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:42.841593 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.841643 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 09:16:42.843310 (+ 1667us) client_negotiation.cc:240] Negotiation successful
0925 09:16:42.843507 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5720,"thread_start_us":12500,"threads_started":1}
I20260925 09:16:42.849989 16633 raft_consensus.cc:686] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [term 1 LEADER]: Becoming Leader. State: Replica: 9c78be255e794292b8e2d7a74cf73672, State: Running, Role: LEADER
I20260925 09:16:42.845532 16279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deedc4dc6eee468e891184587d592f82" candidate_uuid: "9c78be255e794292b8e2d7a74cf73672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851853ad1a84031b9010eb715194fca" is_pre_election: true
I20260925 09:16:42.850587 16633 consensus_queue.cc:237] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54fca7eb395b409a99e7018ebe49a1fd" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 46539 } } peers { permanent_uuid: "2851853ad1a84031b9010eb715194fca" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34231 } } peers { permanent_uuid: "9c78be255e794292b8e2d7a74cf73672" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34735 } }
I20260925 09:16:42.851583 16278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deedc4dc6eee468e891184587d592f82" candidate_uuid: "9c78be255e794292b8e2d7a74cf73672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851853ad1a84031b9010eb715194fca"
I20260925 09:16:42.852102 16471 heartbeater.cc:503] Master 127.3.19.126:46341 was elected leader, sending a full tablet report...
W20260925 09:16:42.852417 16494 leader_election.cc:341] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2851853ad1a84031b9010eb715194fca (127.3.19.65:34231): Illegal state: must be running to vote when last-logged opid is not known
W20260925 09:16:42.853089 16494 leader_election.cc:341] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2851853ad1a84031b9010eb715194fca (127.3.19.65:34231): Illegal state: must be running to vote when last-logged opid is not known
I20260925 09:16:42.859690 16632 tablet_bootstrap.cc:492] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca: No bootstrap required, opened a new log
I20260925 09:16:42.860006 16632 ts_tablet_manager.cc:1397] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.006s
I20260925 09:16:42.868011 16628 ts_tablet_manager.cc:1428] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd: Time spent starting tablet: real 0.089s user 0.029s sys 0.015s
I20260925 09:16:42.868506 16124 catalog_manager.cc:5495] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c78be255e794292b8e2d7a74cf73672 (127.3.19.67). New cstate: current_term: 1 leader_uuid: "9c78be255e794292b8e2d7a74cf73672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54fca7eb395b409a99e7018ebe49a1fd" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 46539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851853ad1a84031b9010eb715194fca" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c78be255e794292b8e2d7a74cf73672" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34735 } health_report { overall_health: HEALTHY } } }
I20260925 09:16:42.875816 16632 raft_consensus.cc:348] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54fca7eb395b409a99e7018ebe49a1fd" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 46539 } } peers { permanent_uuid: "2851853ad1a84031b9010eb715194fca" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34231 } } peers { permanent_uuid: "9c78be255e794292b8e2d7a74cf73672" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34735 } }
I20260925 09:16:42.876173 16632 raft_consensus.cc:374] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:42.876292 16632 raft_consensus.cc:729] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2851853ad1a84031b9010eb715194fca, State: Initialized, Role: FOLLOWER
I20260925 09:16:42.876916 16632 consensus_queue.cc:260] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54fca7eb395b409a99e7018ebe49a1fd" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 46539 } } peers { permanent_uuid: "2851853ad1a84031b9010eb715194fca" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34231 } } peers { permanent_uuid: "9c78be255e794292b8e2d7a74cf73672" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34735 } }
I20260925 09:16:42.880030 16632 ts_tablet_manager.cc:1428] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca: Time spent starting tablet: real 0.020s user 0.006s sys 0.003s
I20260925 09:16:42.888054 16325 heartbeater.cc:503] Master 127.3.19.126:46341 was elected leader, sending a full tablet report...
I20260925 09:16:42.913980 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.908018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35838 (local address 127.3.19.67:34735)
0925 09:16:42.908201 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:42.908208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:42.908325 (+ 117us) server_negotiation.cc:407] Connection header received
0925 09:16:42.908559 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:42.908564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:42.908647 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:42.908800 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:16:42.909161 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.910030 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.911629 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.912060 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.912198 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:42.912726 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:16:42.913203 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:16:42.913503 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:42.913550 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 09:16:42.913621 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 09:16:42.916162 16326 tablet.cc:2369] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:16:42.922246 16279 raft_consensus.cc:3037] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:42.922734 16424 raft_consensus.cc:1261] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd [term 1 FOLLOWER]: Refusing update from remote peer 9c78be255e794292b8e2d7a74cf73672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:16:42.923794 16633 consensus_queue.cc:1035] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54fca7eb395b409a99e7018ebe49a1fd" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 46539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:16:42.925626 16279 raft_consensus.cc:1261] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca [term 1 FOLLOWER]: Refusing update from remote peer 9c78be255e794292b8e2d7a74cf73672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:16:42.926530 16633 consensus_queue.cc:1035] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851853ad1a84031b9010eb715194fca" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:16:42.943643 16648 mvcc.cc:205] Tried to move back new op lower bound from 7333182680760877056 to 7333182680483745792. Current Snapshot: MvccSnapshot[applied={T|T < 7333182680760877056}]
I20260925 09:16:42.944729 16649 mvcc.cc:205] Tried to move back new op lower bound from 7333182680760877056 to 7333182680483745792. Current Snapshot: MvccSnapshot[applied={T|T < 7333182680760877056}]
I20260925 09:16:42.948462 16650 mvcc.cc:205] Tried to move back new op lower bound from 7333182680760877056 to 7333182680483745792. Current Snapshot: MvccSnapshot[applied={T|T < 7333182680760877056}]
I20260925 09:16:42.972464 16125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:16:42.972762 16125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:16:42.977164 16125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eef4711ed1144c2a8b200d496d3699b5: Sending DeleteTablet for 3 replicas of tablet deedc4dc6eee468e891184587d592f82
I20260925 09:16:42.978430 16259 tablet_service.cc:1514] Processing DeleteTablet for tablet deedc4dc6eee468e891184587d592f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:16:42 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:35810
I20260925 09:16:42.978439 16404 tablet_service.cc:1514] Processing DeleteTablet for tablet deedc4dc6eee468e891184587d592f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:16:42 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:45634
I20260925 09:16:42.979691 16541 tablet_service.cc:1514] Processing DeleteTablet for tablet deedc4dc6eee468e891184587d592f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:16:42 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:35836
I20260925 09:16:42.980931 16659 tablet_replica.cc:330] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd: stopping tablet replica
I20260925 09:16:42.981099 16658 tablet_replica.cc:330] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca: stopping tablet replica
I20260925 09:16:42.981391 16659 raft_consensus.cc:2229] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:16:42.981721 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 16163
I20260925 09:16:42.981873 16659 raft_consensus.cc:2258] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:16:42.984057 16659 ts_tablet_manager.cc:1905] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:16:42.984249 16658 raft_consensus.cc:2229] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:16:42.984685 16658 raft_consensus.cc:2258] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:16:42.989888 16660 tablet_replica.cc:330] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672: stopping tablet replica
I20260925 09:16:42.990404 16660 raft_consensus.cc:2229] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:16:42.991184 16660 raft_consensus.cc:2258] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:16:42.992194 16659 ts_tablet_manager.cc:1918] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:16:42.992353 16659 log.cc:1196] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/deedc4dc6eee468e891184587d592f82
I20260925 09:16:42.992933 16659 ts_tablet_manager.cc:1939] T deedc4dc6eee468e891184587d592f82 P 54fca7eb395b409a99e7018ebe49a1fd: Deleting consensus metadata
I20260925 09:16:42.993875 16660 ts_tablet_manager.cc:1905] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:16:42.994858 16112 catalog_manager.cc:4841] TS 54fca7eb395b409a99e7018ebe49a1fd (127.3.19.66:46539): tablet deedc4dc6eee468e891184587d592f82 (table test-table [id=50aaa3e150484f98afc7014a8553b13a]) successfully deleted
I20260925 09:16:43.003199 16660 ts_tablet_manager.cc:1918] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:16:43.003486 16660 log.cc:1196] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/deedc4dc6eee468e891184587d592f82
I20260925 09:16:43.004185 16660 ts_tablet_manager.cc:1939] T deedc4dc6eee468e891184587d592f82 P 9c78be255e794292b8e2d7a74cf73672: Deleting consensus metadata
I20260925 09:16:43.005815 16658 ts_tablet_manager.cc:1905] T deedc4dc6eee468e891184587d592f82 P 2851853ad1a84031b9010eb715194fca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:16:43.006116 16111 catalog_manager.cc:4841] TS 9c78be255e794292b8e2d7a74cf73672 (127.3.19.67:34735): tablet deedc4dc6eee468e891184587d592f82 (table test-table [id=50aaa3e150484f98afc7014a8553b13a]) successfully deleted
I20260925 09:16:43.009892 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:42.984967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.3.19.65:34231)
0925 09:16:42.985178 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:42.985187 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:42.985210 (+ 23us) server_negotiation.cc:407] Connection header received
0925 09:16:42.985300 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:42.985308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:42.985403 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:42.985520 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:42.990927 (+ 5407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.992051 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.995998 (+ 3947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:42.996626 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:42.996753 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:43.000408 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:43.000427 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:43.000447 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:43.000500 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:43.004724 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:43.005691 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:43.005700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:43.005704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:43.005878 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:43.006743 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:43.006751 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:43.006757 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:43.007298 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:43.007687 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:43.009010 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0925 09:16:43.009349 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 09:16:43.010921 16320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:43.161937 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 16163
I20260925 09:16:43.179867 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 16329
W20260925 09:16:43.179833 16109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.19.65:34231 (error 108)
I20260925 09:16:43.182070 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:43.181543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34231 (local address 127.0.0.1:35822)
0925 09:16:43.181756 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:43.181879 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 09:16:43.182282 16109 catalog_manager.cc:4584] TS 2851853ad1a84031b9010eb715194fca (127.3.19.65:34231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet deedc4dc6eee468e891184587d592f82: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34231: connect: Connection refused (error 111)
I20260925 09:16:43.195755 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:43.180658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.3.19.66:46539)
0925 09:16:43.180857 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:43.180866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:43.181121 (+ 255us) server_negotiation.cc:407] Connection header received
0925 09:16:43.181449 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:43.181456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:43.181529 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:43.181651 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:43.183422 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:43.184516 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:43.185848 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:43.186463 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:43.186589 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:43.189052 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:43.189069 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:43.189087 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:43.189127 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:43.192891 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:43.193646 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:43.193656 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:43.193660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:43.193828 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:43.194181 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:43.194186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:43.194189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:43.194618 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:43.194914 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:43.195109 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 09:16:43.195340 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 09:16:43.196494 16466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:43.235965 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:43.235431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34231 (local address 127.0.0.1:35832)
0925 09:16:43.235684 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:43.235796 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 09:16:43.237111 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:43.236676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34231 (local address 127.0.0.1:35848)
0925 09:16:43.236846 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:43.236943 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 09:16:43.285444 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 16329
I20260925 09:16:43.297806 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:43.297265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34231 (local address 127.0.0.1:35850)
0925 09:16:43.297503 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:43.297612 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 09:16:43.299074 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:43.298606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34231 (local address 127.0.0.1:35852)
0925 09:16:43.298793 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:43.298895 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 09:16:43.302814 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 16475
I20260925 09:16:43.316638 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:43.303633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.3.19.67:34735)
0925 09:16:43.303814 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:43.303820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:43.303843 (+ 23us) server_negotiation.cc:407] Connection header received
0925 09:16:43.304017 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:43.304022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:43.304081 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:43.304185 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:43.305462 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:43.306305 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:43.307641 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:43.308173 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:43.308308 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:43.310912 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:43.310922 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:43.310935 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:43.310969 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:43.313874 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:43.314543 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:43.314550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:43.314552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:43.314691 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:43.315067 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:43.315073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:43.315077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:43.315423 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:43.315790 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:43.316070 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 09:16:43.316293 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 09:16:43.317562 16601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:43.379846 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:43.379301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34231 (local address 127.0.0.1:35856)
0925 09:16:43.379567 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:43.379679 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 09:16:43.380975 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:43.380516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34231 (local address 127.0.0.1:35864)
0925 09:16:43.380717 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:43.380813 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 09:16:43.434989 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 16475
I20260925 09:16:43.454969 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 16092
I20260925 09:16:43.456051 16156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:43.566612 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:43.565984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34231 (local address 127.0.0.1:35878)
0925 09:16:43.566162 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:43.566258 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 09:16:43.566992 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 16092
I20260925 09:16:43.592854 3149 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 09:16:43.927369 3149 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 09:16:43.927562 3149 mini_ranger.cc:66] Stopping Ranger...
I20260925 09:16:44.272048 3149 mini_ranger.cc:68] Stopped Ranger
2026-09-25 09:16:44.272 UTC [15140] LOG: received smart shutdown request
2026-09-25 09:16:44.276 UTC [15140] LOG: background worker "logical replication launcher" (PID 15149) exited with exit code 1
2026-09-25 09:16:44.276 UTC [15144] LOG: shutting down
2026-09-25 09:16:44.432 UTC [15140] LOG: database system is shut down
2026-09-25T09:16:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51701 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:16:44 dist-test-slave-l6qh krb5kdc[16675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:16:44 dist-test-slave-l6qh krb5kdc[16675](info): set up 2 sockets
Sep 25 09:16:44 dist-test-slave-l6qh krb5kdc[16675](info): commencing operation
krb5kdc: starting...
W20260925 09:16:45.646744 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:16:45 dist-test-slave-l6qh krb5kdc[16675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:16:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:16:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:16:45.890682 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46817
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:43691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:46817,127.3.19.125:41915,127.3.19.124:39149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:16:46.217044 16688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:46.217412 16688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:46.217586 16688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:46.224632 16688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:16:46.224758 16688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:46.224849 16688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:46.224949 16688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:16:46.225032 16688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:16:46.234748 16688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:43691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:46817,127.3.19.125:41915,127.3.19.124:39149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:46.236058 16688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:46.237449 16688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:46.249570 16696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:46.249665 16694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:46.249634 16693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:46.250221 16688 server_base.cc:1034] running on GCE node
I20260925 09:16:46.251009 16688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:46.252437 16688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:46.253669 16688 hybrid_clock.cc:648] HybridClock initialized: now 1790327806253622 us; error 52 us; skew 500 ppm
I20260925 09:16:46.254009 16688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:46 dist-test-slave-l6qh krb5kdc[16675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327806, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:46.256986 16688 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:16:46.258488 16688 webserver.cc:460] Webserver started at http://127.3.19.126:38797/ using document root <none> and password file <none>
I20260925 09:16:46.259107 16688 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:46.259202 16688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:46.259445 16688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:16:46.262375 16688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "da3736482672495192ab4edcb36a50cd"
format_stamp: "Formatted at 2026-09-25 09:16:46 on dist-test-slave-l6qh"
server_key: "7532caa58a509cd5534a00f5b179bc0d"
server_key_iv: "c42300d7ac3db2c2318571cdd74d5a2e"
server_key_version: "encryptionkey@0"
I20260925 09:16:46.263113 16688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "da3736482672495192ab4edcb36a50cd"
format_stamp: "Formatted at 2026-09-25 09:16:46 on dist-test-slave-l6qh"
server_key: "7532caa58a509cd5534a00f5b179bc0d"
server_key_iv: "c42300d7ac3db2c2318571cdd74d5a2e"
server_key_version: "encryptionkey@0"
I20260925 09:16:46.268092 16688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:16:46.271430 16703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:46.272737 16688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:16:46.272918 16688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "da3736482672495192ab4edcb36a50cd"
format_stamp: "Formatted at 2026-09-25 09:16:46 on dist-test-slave-l6qh"
server_key: "7532caa58a509cd5534a00f5b179bc0d"
server_key_iv: "c42300d7ac3db2c2318571cdd74d5a2e"
server_key_version: "encryptionkey@0"
I20260925 09:16:46.273128 16688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:46.286450 16688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:46.290354 16688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:46.290637 16688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:46.309309 16688 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:46817
I20260925 09:16:46.309334 16754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:46817 every 8 connection(s)
I20260925 09:16:46.310715 16688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:16:46.312763 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 16688
I20260925 09:16:46.312979 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:16:46.313730 3149 external_mini_cluster.cc:1428] Setting key 5f18e08fa07ab6ff79602adf9b539627
I20260925 09:16:46.314646 16755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:46.320976 16755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.19.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.125@KRBTEST.COM" created.
Sep 25 09:16:46 dist-test-slave-l6qh krb5kdc[16675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327806, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:46.348001 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.324522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46817 (local address 127.0.0.1:53218)
0925 09:16:46.325744 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.325802 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:46.327503 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:46.328421 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:46.328426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:46.328947 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:46.329332 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:46.329351 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:46.331684 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:46.331688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:46.332712 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:46.332722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:46.333999 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:46.334004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:46.334132 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:46.334823 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:46.334843 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:46.338861 (+ 4018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:46.343296 (+ 4435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:46.343305 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:46.343315 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:46.343737 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:46.344216 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:46.344219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:46.344222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:46.344366 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:46.345413 (+ 1047us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:46.345419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:46.345625 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 09:16:46.345926 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 09:16:46.346611 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":223,"spinlock_wait_cycles":77184,"thread_start_us":778,"threads_started":1}
I20260925 09:16:46.348034 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.324522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.3.19.126:46817)
0925 09:16:46.325360 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:46.325375 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:46.325942 (+ 567us) server_negotiation.cc:407] Connection header received
0925 09:16:46.327863 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:46.327872 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:46.328174 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:46.328386 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:46.329521 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:46.331521 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:46.332853 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:46.333797 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:46.333981 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:46.339112 (+ 5131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:46.339124 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:46.339143 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:46.339186 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:46.343161 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:46.343881 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:46.343892 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:46.343898 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:46.344104 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:46.344489 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:46.344496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:46.344502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:46.345276 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:46.345486 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:46.346067 (+ 581us) server_negotiation.cc:299] Negotiation successful
0925 09:16:46.346494 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":565,"thread_start_us":361,"threads_started":1}
I20260925 09:16:46.351399 16755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } has no permanent_uuid. Determining permanent_uuid...
I20260925 09:16:46.352509 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.352020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:41915 (local address 127.0.0.1:48594)
0925 09:16:46.352214 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.352346 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 09:16:46.352769 16704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
I20260925 09:16:46.354552 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.354098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:41915 (local address 127.0.0.1:48598)
0925 09:16:46.354274 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.354366 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 09:16:46.354759 16755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:41915: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.125.keytab.
Entry for principal kudu/127.3.19.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.125.keytab.
I20260925 09:16:46.369395 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.125:41915
--webserver_interface=127.3.19.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.125.keytab
--principal=kudu/127.3.19.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:43691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:46817,127.3.19.125:41915,127.3.19.124:39149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:16:46.404155 16755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } attempt: 1
I20260925 09:16:46.405400 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.404859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:41915 (local address 127.0.0.1:48608)
0925 09:16:46.405094 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.405220 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 09:16:46.406461 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.406046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:41915 (local address 127.0.0.1:48610)
0925 09:16:46.406239 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.406320 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 09:16:46.406675 16755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:41915: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
I20260925 09:16:46.474967 16755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } attempt: 2
I20260925 09:16:46.476183 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.475684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:41915 (local address 127.0.0.1:48622)
0925 09:16:46.475948 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.476048 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 09:16:46.477339 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.476952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:41915 (local address 127.0.0.1:48626)
0925 09:16:46.477101 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.477162 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 09:16:46.477643 16755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:41915: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
I20260925 09:16:46.568915 16755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } attempt: 3
I20260925 09:16:46.570185 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.569740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:41915 (local address 127.0.0.1:48632)
0925 09:16:46.569957 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.570047 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 09:16:46.571192 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.570839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:41915 (local address 127.0.0.1:48648)
0925 09:16:46.570975 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.571072 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 09:16:46.571376 16755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:41915: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
W20260925 09:16:46.699175 16763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:46.699539 16763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:46.699692 16763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:46.706820 16763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:16:46.706959 16763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:46.707059 16763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:46.707163 16763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:16:46.707252 16763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:16:46.714586 16755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } attempt: 4
I20260925 09:16:46.715767 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.715246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:41915 (local address 127.0.0.1:48662)
0925 09:16:46.715535 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.715630 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260925 09:16:46.716732 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.716393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:41915 (local address 127.0.0.1:48678)
0925 09:16:46.716544 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.716620 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 09:16:46.716919 16755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:41915: Network error: Client connection negotiation failed: client connection to 127.3.19.125:41915: connect: Connection refused (error 111)
I20260925 09:16:46.717546 16763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:43691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:46817,127.3.19.125:41915,127.3.19.124:39149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.125.keytab
--principal=kudu/127.3.19.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.125:41915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.19.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:46.719112 16763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:46.720629 16763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:46.732748 16769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:46.732724 16771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:46.732894 16768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:46.733560 16763 server_base.cc:1034] running on GCE node
I20260925 09:16:46.734246 16763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:46.735466 16763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:46.736709 16763 hybrid_clock.cc:648] HybridClock initialized: now 1790327806736643 us; error 38 us; skew 500 ppm
I20260925 09:16:46.737043 16763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:46 dist-test-slave-l6qh krb5kdc[16675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327806, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:46.740120 16763 init.cc:377] Logged in from keytab as kudu/127.3.19.125@KRBTEST.COM (short username kudu)
I20260925 09:16:46.741954 16763 webserver.cc:460] Webserver started at http://127.3.19.125:43025/ using document root <none> and password file <none>
I20260925 09:16:46.742609 16763 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:46.742756 16763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:46.743059 16763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:16:46.746024 16763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/data/instance:
uuid: "f1ae1ed863cd4b7894b39a889e75e685"
format_stamp: "Formatted at 2026-09-25 09:16:46 on dist-test-slave-l6qh"
server_key: "a75b8c7f467e47a6bf7858d19cdfadc6"
server_key_iv: "f044b915b4e605fb7b0202847ae9306c"
server_key_version: "encryptionkey@0"
I20260925 09:16:46.746801 16763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/wal/instance:
uuid: "f1ae1ed863cd4b7894b39a889e75e685"
format_stamp: "Formatted at 2026-09-25 09:16:46 on dist-test-slave-l6qh"
server_key: "a75b8c7f467e47a6bf7858d19cdfadc6"
server_key_iv: "f044b915b4e605fb7b0202847ae9306c"
server_key_version: "encryptionkey@0"
I20260925 09:16:46.751933 16763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:16:46.755373 16778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:46.756599 16763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:16:46.756790 16763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/wal
uuid: "f1ae1ed863cd4b7894b39a889e75e685"
format_stamp: "Formatted at 2026-09-25 09:16:46 on dist-test-slave-l6qh"
server_key: "a75b8c7f467e47a6bf7858d19cdfadc6"
server_key_iv: "f044b915b4e605fb7b0202847ae9306c"
server_key_version: "encryptionkey@0"
I20260925 09:16:46.756983 16763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:46.805478 16763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:46.810026 16763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:46.810325 16763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:46.828778 16763 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.125:41915
I20260925 09:16:46.828814 16829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.125:41915 every 8 connection(s)
I20260925 09:16:46.830163 16763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/data/info.pb
I20260925 09:16:46.830890 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 16763
I20260925 09:16:46.831092 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-1/wal/instance
I20260925 09:16:46.831626 3149 external_mini_cluster.cc:1428] Setting key 8d71a6556c546d8c955272fbb6f587ec
I20260925 09:16:46.834030 16830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:46.840389 16830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.19.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.124@KRBTEST.COM" created.
Sep 25 09:16:46 dist-test-slave-l6qh krb5kdc[16675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327806, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.125@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:46.861959 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.843046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.3.19.126:46817)
0925 09:16:46.843220 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:46.843224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:46.844615 (+ 1391us) server_negotiation.cc:407] Connection header received
0925 09:16:46.846514 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:46.846520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:46.846612 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:46.846728 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:46.848216 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:46.848877 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:46.850432 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:46.850862 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:46.851005 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:46.855567 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:46.855571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:46.855574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:46.855608 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:46.858261 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:46.859210 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:46.859214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:46.859216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:46.859327 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:46.859846 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:46.859849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:46.859851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:46.860197 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:46.860345 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:46.861298 (+ 953us) server_negotiation.cc:299] Negotiation successful
0925 09:16:46.861581 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 09:16:46.862571 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.843607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46817 (local address 127.0.0.1:53226)
0925 09:16:46.844406 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.844459 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:46.846242 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:46.846972 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:46.846980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:46.847478 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:46.848050 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:46.848066 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:46.849044 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:46.849047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:46.850283 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:46.850295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:46.851078 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:46.851085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:46.851371 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:46.852087 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:46.852123 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:46.855329 (+ 3206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:46.858406 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:46.858417 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:46.858434 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:46.859065 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:46.859489 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:46.859497 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:46.859503 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:46.859730 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:46.860331 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:46.860341 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:46.860794 (+ 453us) client_negotiation.cc:769] Sending connection context
0925 09:16:46.861119 (+ 325us) client_negotiation.cc:240] Negotiation successful
0925 09:16:46.861635 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":311,"threads_started":1}
I20260925 09:16:46.864200 16830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:16:46 dist-test-slave-l6qh krb5kdc[16675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327806, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.125@KRBTEST.COM for kudu/127.3.19.125@KRBTEST.COM
I20260925 09:16:46.880872 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.864805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:41915 (local address 127.0.0.1:48690)
0925 09:16:46.864977 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.864994 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:46.865089 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:46.866427 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:46.866433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:46.866958 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:46.867103 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:46.867113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:46.868647 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:46.868655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:46.869699 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:46.869712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:46.870657 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:46.870665 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:46.870901 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:46.871522 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:46.871542 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:46.874009 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:46.877784 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:46.877787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:46.877790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:46.878214 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:46.878697 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:46.878702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:46.878707 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:46.878885 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:46.879661 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:46.879669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:46.879839 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 09:16:46.880029 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 09:16:46.880328 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 09:16:46.881297 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.865159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.3.19.125:41915)
0925 09:16:46.865938 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:46.865945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:46.865977 (+ 32us) server_negotiation.cc:407] Connection header received
0925 09:16:46.866027 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:46.866033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:46.866237 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:46.866435 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:46.867284 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:46.868476 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:46.869857 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:46.870452 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:46.870627 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:46.874265 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:46.874277 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:46.874290 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:46.874326 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:46.877614 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:46.878333 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:46.878340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:46.878342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:46.878526 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:46.879035 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:46.879041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:46.879045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:46.879447 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:46.879779 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:46.880263 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 09:16:46.880489 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":433,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.124.keytab.
Entry for principal kudu/127.3.19.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.124.keytab.
I20260925 09:16:46.883842 16830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } has no permanent_uuid. Determining permanent_uuid...
I20260925 09:16:46.884447 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.124:39149
--webserver_interface=127.3.19.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.124.keytab
--principal=kudu/127.3.19.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:43691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:46817,127.3.19.125:41915,127.3.19.124:39149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:16:46.885021 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.884534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59218)
0925 09:16:46.884760 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.884892 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260925 09:16:46.885394 16780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
I20260925 09:16:46.887027 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.886718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59234)
0925 09:16:46.886852 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.886932 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 09:16:46.887341 16830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:39149: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
I20260925 09:16:46.936708 16830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } attempt: 1
I20260925 09:16:46.938011 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.937460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59238)
0925 09:16:46.937675 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.937827 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 09:16:46.939046 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:46.938646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59250)
0925 09:16:46.938819 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:46.938906 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 09:16:46.939266 16830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:39149: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
I20260925 09:16:47.007513 16830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } attempt: 2
I20260925 09:16:47.008832 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.008207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59254)
0925 09:16:47.008480 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.008610 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 09:16:47.009936 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.009567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59268)
0925 09:16:47.009717 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.009802 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 09:16:47.010155 16830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:39149: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
I20260925 09:16:47.016270 16755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } attempt: 5
Sep 25 09:16:47 dist-test-slave-l6qh krb5kdc[16675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327806, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for kudu/127.3.19.125@KRBTEST.COM
I20260925 09:16:47.029608 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.016973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48698 (local address 127.3.19.125:41915)
0925 09:16:47.017126 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:47.017132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:47.017214 (+ 82us) server_negotiation.cc:407] Connection header received
0925 09:16:47.017452 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:47.017461 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:47.017533 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:47.017634 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:47.018384 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.019088 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.020285 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.020737 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.020868 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.024341 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:47.024345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:47.024348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:47.024379 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:47.026954 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.027845 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.027849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.027851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.027975 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.028377 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.028379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.028381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.028631 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:47.028767 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:47.029202 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 09:16:47.029419 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 09:16:47.029736 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.016904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:41915 (local address 127.0.0.1:48698)
0925 09:16:47.017101 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.017120 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:47.017313 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:47.017663 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:47.017668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:47.018064 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:47.018240 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:47.018249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.019218 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.019223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:47.020136 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:47.020155 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.020890 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.020895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:47.021062 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.021704 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:47.021722 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:47.024167 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:47.027109 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:47.027116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:47.027122 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:47.027733 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:47.028086 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:47.028091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:47.028094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:47.028275 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:47.028776 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:47.028781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:47.028951 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 09:16:47.029128 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 09:16:47.029416 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 09:16:47.030866 16755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } has no permanent_uuid. Determining permanent_uuid...
I20260925 09:16:47.032008 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.031586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59276)
0925 09:16:47.031759 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.031845 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 09:16:47.032922 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.032583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59288)
0925 09:16:47.032732 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.032807 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 09:16:47.033154 16755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:39149: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
I20260925 09:16:47.084439 16755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } attempt: 1
I20260925 09:16:47.085829 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.085153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59290)
0925 09:16:47.085448 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.085587 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260925 09:16:47.087030 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.086576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59304)
0925 09:16:47.086760 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.086873 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 09:16:47.087285 16755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:39149: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
I20260925 09:16:47.101331 16830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } attempt: 3
I20260925 09:16:47.102491 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.101930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59306)
0925 09:16:47.102177 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.102294 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 09:16:47.103654 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.103145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59320)
0925 09:16:47.103394 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.103495 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260925 09:16:47.103864 16830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:39149: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
I20260925 09:16:47.155520 16755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } attempt: 2
I20260925 09:16:47.156844 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.156313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59322)
0925 09:16:47.156547 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.156653 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 09:16:47.157953 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.157646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59336)
0925 09:16:47.157791 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.157854 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 09:16:47.158176 16755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:39149: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
W20260925 09:16:47.213469 16837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:47.213776 16837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:47.213932 16837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:47.221180 16837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:16:47.221370 16837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:47.221460 16837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:47.221545 16837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:16:47.221606 16837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:16:47.231057 16837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:43691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:46817,127.3.19.125:41915,127.3.19.124:39149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.124.keytab
--principal=kudu/127.3.19.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.124:39149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.19.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:47.232352 16837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:47.233659 16837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:47.245316 16842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:47.245280 16845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:47.245642 16843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:47.246109 16837 server_base.cc:1034] running on GCE node
I20260925 09:16:47.246918 16837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:47.247092 16830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } attempt: 4
I20260925 09:16:47.248148 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.247711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59352)
0925 09:16:47.247915 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.247994 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 09:16:47.248881 16837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:47.249152 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.248797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59368)
0925 09:16:47.248969 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.249052 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 09:16:47.249414 16830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:39149: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
I20260925 09:16:47.250089 16837 hybrid_clock.cc:648] HybridClock initialized: now 1790327807250031 us; error 52 us; skew 500 ppm
I20260925 09:16:47.250435 16837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:47 dist-test-slave-l6qh krb5kdc[16675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:47.253461 16837 init.cc:377] Logged in from keytab as kudu/127.3.19.124@KRBTEST.COM (short username kudu)
I20260925 09:16:47.254858 16837 webserver.cc:460] Webserver started at http://127.3.19.124:39793/ using document root <none> and password file <none>
I20260925 09:16:47.255463 16837 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:47.255582 16837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:47.255899 16837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:16:47.258889 16837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/data/instance:
uuid: "3bfc3ff243e34aa69029f9782ee16910"
format_stamp: "Formatted at 2026-09-25 09:16:47 on dist-test-slave-l6qh"
server_key: "c4d08066da151ef6c6bed2c1be342604"
server_key_iv: "855d094ace8a3fd8886dbff3404bb70b"
server_key_version: "encryptionkey@0"
I20260925 09:16:47.259609 16837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/wal/instance:
uuid: "3bfc3ff243e34aa69029f9782ee16910"
format_stamp: "Formatted at 2026-09-25 09:16:47 on dist-test-slave-l6qh"
server_key: "c4d08066da151ef6c6bed2c1be342604"
server_key_iv: "855d094ace8a3fd8886dbff3404bb70b"
server_key_version: "encryptionkey@0"
I20260925 09:16:47.264415 16755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } attempt: 3
I20260925 09:16:47.264765 16837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 09:16:47.265450 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.264996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59372)
0925 09:16:47.265182 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.265313 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 09:16:47.266463 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.266128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59376)
0925 09:16:47.266306 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.266374 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260925 09:16:47.266734 16755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:39149: Network error: Client connection negotiation failed: client connection to 127.3.19.124:39149: connect: Connection refused (error 111)
I20260925 09:16:47.268432 16852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:47.269711 16837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 09:16:47.269884 16837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/wal
uuid: "3bfc3ff243e34aa69029f9782ee16910"
format_stamp: "Formatted at 2026-09-25 09:16:47 on dist-test-slave-l6qh"
server_key: "c4d08066da151ef6c6bed2c1be342604"
server_key_iv: "855d094ace8a3fd8886dbff3404bb70b"
server_key_version: "encryptionkey@0"
I20260925 09:16:47.270056 16837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:47.301844 16837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:47.305992 16837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:47.306274 16837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:47.324934 16837 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.124:39149
I20260925 09:16:47.324965 16903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.124:39149 every 8 connection(s)
I20260925 09:16:47.326284 16837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/data/info.pb
I20260925 09:16:47.327025 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 16837
I20260925 09:16:47.327253 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790327600440376-3149-0/minicluster-data/master-2/wal/instance
I20260925 09:16:47.327791 3149 external_mini_cluster.cc:1428] Setting key eefaaa4cf03f34dcec94f8eb941e0c2e
I20260925 09:16:47.330605 16904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:47.338572 16904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:16:47 dist-test-slave-l6qh krb5kdc[16675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.124@KRBTEST.COM
Sep 25 09:16:47 dist-test-slave-l6qh krb5kdc[16675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.124@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:47.351358 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.329570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.3.19.124:39149)
0925 09:16:47.330311 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:47.330322 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:47.330361 (+ 39us) server_negotiation.cc:407] Connection header received
0925 09:16:47.331977 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:47.331997 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:47.332334 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:47.332626 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:47.334392 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.335657 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.337641 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.338222 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.338339 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.343398 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:47.343414 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:47.343427 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:47.343477 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:47.347003 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.347843 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.347849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.347854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.348055 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.348463 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.348468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.348471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.349181 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:47.349502 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:47.349852 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 09:16:47.350272 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":351,"threads_started":1}
I20260925 09:16:47.357746 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.341625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46817 (local address 127.0.0.1:53240)
0925 09:16:47.342409 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.342461 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:47.342664 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:47.343068 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:47.343073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:47.343857 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:47.344209 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:47.344247 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.345433 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.345441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:47.347155 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:47.347166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.348135 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.348141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:47.348285 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.348858 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:47.348878 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:47.351253 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:47.354846 (+ 3593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:47.354861 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:47.354870 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:47.355626 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:47.356074 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:47.356080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:47.356084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:47.356219 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:47.356864 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:47.356870 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:47.357006 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 09:16:47.357163 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 09:16:47.357463 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":416,"threads_started":1}
I20260925 09:16:47.357949 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.341648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.3.19.126:46817)
0925 09:16:47.341848 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:47.341854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:47.342568 (+ 714us) server_negotiation.cc:407] Connection header received
0925 09:16:47.342809 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:47.342816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:47.342902 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:47.343012 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:47.344405 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.345266 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.347289 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.347943 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.348044 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.351464 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:47.351470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:47.351474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:47.351526 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:47.354705 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.355796 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.355800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.355802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.355952 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.356337 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.356343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.356345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.356735 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:47.356900 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:47.357350 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 09:16:47.357587 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 09:16:47.359337 16904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:16:47 dist-test-slave-l6qh krb5kdc[16675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
Sep 25 09:16:47 dist-test-slave-l6qh krb5kdc[16675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.124@KRBTEST.COM for kudu/127.3.19.125@KRBTEST.COM
I20260925 09:16:47.371536 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.354608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.3.19.126:46817)
0925 09:16:47.356247 (+ 1639us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:47.356273 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:47.356300 (+ 27us) server_negotiation.cc:407] Connection header received
0925 09:16:47.356354 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:47.356362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:47.356458 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:47.356548 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:47.358274 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.359334 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.360733 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.361378 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.361523 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.365930 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:47.365939 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:47.365943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:47.365976 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:47.368901 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.369577 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.369584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.369587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.369701 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.370047 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.370052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.370057 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.370346 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:47.370618 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:47.370910 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 09:16:47.371133 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1527,"thread_start_us":1332,"threads_started":1}
I20260925 09:16:47.372798 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.359957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:41915 (local address 127.0.0.1:48700)
0925 09:16:47.360339 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.360356 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:47.360482 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:47.360923 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:47.360927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:47.361374 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:47.361549 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:47.361560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.362946 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.362950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:47.363789 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:47.363799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.364708 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.364715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:47.364940 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.365542 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:47.365559 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:47.367595 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:47.370401 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:47.370409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:47.370413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:47.370768 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:47.371160 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:47.371164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:47.371168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:47.371351 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:47.371923 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:47.371928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:47.372070 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 09:16:47.372217 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 09:16:47.372467 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291}
I20260925 09:16:47.372865 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.360136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.3.19.125:41915)
0925 09:16:47.360331 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:47.360337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:47.360585 (+ 248us) server_negotiation.cc:407] Connection header received
0925 09:16:47.360703 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:47.360710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:47.360789 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:47.360875 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:47.361743 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.362793 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.363927 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.364553 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.364646 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.367782 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:47.367786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:47.367789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:47.367820 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:47.370267 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.370890 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.370902 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.370905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.371032 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.371475 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.371482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.371486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.371804 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:47.371965 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:47.372302 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 09:16:47.372498 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 09:16:47.374054 16904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:16:47 dist-test-slave-l6qh krb5kdc[16675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.124@KRBTEST.COM for kudu/127.3.19.124@KRBTEST.COM
Sep 25 09:16:47 dist-test-slave-l6qh krb5kdc[16675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.125@KRBTEST.COM
I20260925 09:16:47.387075 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.374853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59392)
0925 09:16:47.375004 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.375020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:47.375103 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:47.375473 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:47.375477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:47.375808 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:47.375949 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:47.375956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.376931 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.376939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:47.377874 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:47.377909 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.378828 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.378832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:47.378944 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.379482 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:47.379498 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:47.381544 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:47.384593 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:47.384598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:47.384602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:47.384950 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:47.385374 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:47.385382 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:47.385385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:47.385543 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:47.386131 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:47.386139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:47.386291 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 09:16:47.386409 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 09:16:47.386664 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 09:16:47.387256 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.375060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59392 (local address 127.3.19.124:39149)
0925 09:16:47.375241 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:47.375247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:47.375264 (+ 17us) server_negotiation.cc:407] Connection header received
0925 09:16:47.375303 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:47.375307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:47.375362 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:47.375457 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:47.376149 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.376815 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.378016 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.378427 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.378571 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.381706 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:47.381712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:47.381717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:47.381758 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:47.384460 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.385077 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.385085 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.385089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.385203 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.385651 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.385655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.385659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.386018 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:47.386116 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:47.386526 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 09:16:47.386877 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":59}
I20260925 09:16:47.387872 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.373500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.3.19.125:41915)
0925 09:16:47.373647 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:47.373652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:47.373668 (+ 16us) server_negotiation.cc:407] Connection header received
0925 09:16:47.373713 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:47.373720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:47.373784 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:47.373877 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:47.375248 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.375941 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.377361 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.377776 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.377968 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.382355 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:47.382367 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:47.382371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:47.382400 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:47.384816 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.385654 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.385660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.385663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.385820 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.386230 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.386234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.386236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.386550 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:47.386833 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:47.387145 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 09:16:47.387380 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 09:16:47.392859 16904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910: Bootstrap starting.
I20260925 09:16:47.393023 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:16:47.396026 16904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:47.397028 16904 log.cc:824] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:47.399526 16904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910: No bootstrap required, opened a new log
I20260925 09:16:47.410305 16904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } }
I20260925 09:16:47.410719 16904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:47.410874 16904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfc3ff243e34aa69029f9782ee16910, State: Initialized, Role: FOLLOWER
I20260925 09:16:47.411551 16904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } }
I20260925 09:16:47.415604 16918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } } }
I20260925 09:16:47.416056 16918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:16:47.417160 16904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:16:47.421253 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.402013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.3.19.124:39149)
0925 09:16:47.402387 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:47.402397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:47.402882 (+ 485us) server_negotiation.cc:407] Connection header received
0925 09:16:47.403113 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:47.403118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:47.403184 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:47.403256 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:47.405841 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.406921 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.408874 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.409582 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.409664 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.413912 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:47.413919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:47.413924 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:47.413965 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:47.417727 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.418636 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.418643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.418649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.418829 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.419397 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.419404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.419411 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.419793 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:47.420021 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:47.420605 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 09:16:47.420841 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
I20260925 09:16:47.423760 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.401366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53262 (local address 127.3.19.126:46817)
0925 09:16:47.401595 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:47.401604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:47.402474 (+ 870us) server_negotiation.cc:407] Connection header received
0925 09:16:47.402794 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:47.402803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:47.402884 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:47.402955 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:47.405406 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.406524 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.408744 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.409447 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.409548 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.414278 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:47.414285 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:47.414297 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:47.414336 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:47.418410 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.419893 (+ 1483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.419901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.419907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.420093 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.420690 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.420698 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.420704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.421138 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:47.422796 (+ 1658us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:47.422974 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 09:16:47.423280 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 09:16:47.423811 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.403235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.3.19.125:41915)
0925 09:16:47.403409 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:47.403415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:47.404548 (+ 1133us) server_negotiation.cc:407] Connection header received
0925 09:16:47.404788 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:47.404793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:47.404860 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:47.404946 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:47.407393 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.408443 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.410537 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.411307 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.411450 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.416951 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:47.416957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:47.416962 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:47.417004 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:47.420607 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.421498 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.421504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.421509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.421683 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.422146 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.422151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.422155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.422527 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:47.422772 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:47.423284 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 09:16:47.423515 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 09:16:47.436288 16929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 09:16:47.436481 16929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 09:16:47.443984 16755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } attempt: 4
I20260925 09:16:47.433365 16837 master_runner.cc:186] Error getting master registration for 127.3.19.126:46817: OK, instance_id { permanent_uuid: "da3736482672495192ab4edcb36a50cd" instance_seqno: 1790327806303712 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 09:16:47.444810 16837 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 09:16:47 dist-test-slave-l6qh krb5kdc[16675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327806, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for kudu/127.3.19.124@KRBTEST.COM
I20260925 09:16:47.459327 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.444495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59412)
0925 09:16:47.444913 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.444932 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:47.445030 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:47.445867 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:47.445872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:47.446398 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:47.446557 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:47.446566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.448024 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.448027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:47.448915 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:47.448925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.449882 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.449886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:47.450035 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.450654 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:47.450675 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:47.453373 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:47.457171 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:47.457174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:47.457177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:47.457563 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:47.457929 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:47.457932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:47.457934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:47.458066 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:47.458636 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:47.458639 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:47.458768 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 09:16:47.458871 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 09:16:47.459076 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298}
I20260925 09:16:47.459609 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.445214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.3.19.124:39149)
0925 09:16:47.445421 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:47.445426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:47.445441 (+ 15us) server_negotiation.cc:407] Connection header received
0925 09:16:47.445493 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:47.445498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:47.445603 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:47.445701 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:47.446700 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.447899 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.449048 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.449745 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.449881 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.453530 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:47.453537 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:47.453542 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:47.453584 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:47.457008 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.457665 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.457671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.457674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.457820 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.458163 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.458168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.458172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.458521 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:47.458665 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:47.458987 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 09:16:47.459219 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260925 09:16:47.464727 16755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd: Bootstrap starting.
I20260925 09:16:47.467773 16755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:47.468761 16755 log.cc:824] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:47.471339 16755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd: No bootstrap required, opened a new log
I20260925 09:16:47.479326 16755 raft_consensus.cc:348] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } }
I20260925 09:16:47.479645 16755 raft_consensus.cc:374] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:47.479805 16755 raft_consensus.cc:729] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3736482672495192ab4edcb36a50cd, State: Initialized, Role: FOLLOWER
I20260925 09:16:47.480393 16755 consensus_queue.cc:260] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } }
I20260925 09:16:47.481317 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } } }
I20260925 09:16:47.481715 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:16:47.482467 16755 sys_catalog.cc:564] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [sys.catalog]: configured and running, proceeding with master startup.
W20260925 09:16:47.493922 16944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 09:16:47.494118 16944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 09:16:47.493048 16688 master_runner.cc:186] Error getting master registration for 127.3.19.125:41915: OK, instance_id { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" instance_seqno: 1790327806823382 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 09:16:47.505450 16688 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 09:16:47.518112 16918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:16:47.518500 16918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } }
I20260925 09:16:47.520377 16918 leader_election.cc:288] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3736482672495192ab4edcb36a50cd (127.3.19.126:46817), f1ae1ed863cd4b7894b39a889e75e685 (127.3.19.125:41915)
I20260925 09:16:47.521343 16730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bfc3ff243e34aa69029f9782ee16910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3736482672495192ab4edcb36a50cd" is_pre_election: true
I20260925 09:16:47.521998 16730 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bfc3ff243e34aa69029f9782ee16910 in term 0.
I20260925 09:16:47.523363 16853 leader_election.cc:302] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bfc3ff243e34aa69029f9782ee16910, da3736482672495192ab4edcb36a50cd; no voters:
I20260925 09:16:47.524315 16918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:16:47.524468 16918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:16:47.524564 16918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:47.528029 16918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } }
I20260925 09:16:47.528689 16918 leader_election.cc:288] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [CANDIDATE]: Term 1 election: Requested vote from peers da3736482672495192ab4edcb36a50cd (127.3.19.126:46817), f1ae1ed863cd4b7894b39a889e75e685 (127.3.19.125:41915)
I20260925 09:16:47.529799 16730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bfc3ff243e34aa69029f9782ee16910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3736482672495192ab4edcb36a50cd"
I20260925 09:16:47.530011 16730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:47.533442 16730 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bfc3ff243e34aa69029f9782ee16910 in term 1.
I20260925 09:16:47.533931 16853 leader_election.cc:302] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bfc3ff243e34aa69029f9782ee16910, da3736482672495192ab4edcb36a50cd; no voters:
I20260925 09:16:47.534289 16918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:16:47.535006 16918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [term 1 LEADER]: Becoming Leader. State: Replica: 3bfc3ff243e34aa69029f9782ee16910, State: Running, Role: LEADER
I20260925 09:16:47.535645 16918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } }
I20260925 09:16:47.539593 16946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bfc3ff243e34aa69029f9782ee16910. Latest consensus state: current_term: 1 leader_uuid: "3bfc3ff243e34aa69029f9782ee16910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } } }
I20260925 09:16:47.540019 16946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [sys.catalog]: This master's current role is: LEADER
W20260925 09:16:47.543603 16709 tablet.cc:2369] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:16:47.548610 16830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } attempt: 5
I20260925 09:16:47.551484 16948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:16:47.553902 16948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 09:16:47.558615 16858 tablet.cc:2369] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 09:16:47 dist-test-slave-l6qh krb5kdc[16675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327806, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.125@KRBTEST.COM for kudu/127.3.19.124@KRBTEST.COM
I20260925 09:16:47.575197 16730 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [term 1 FOLLOWER]: Refusing update from remote peer 3bfc3ff243e34aa69029f9782ee16910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:16:47.578410 16946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:16:47.591075 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.549476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:39149 (local address 127.0.0.1:59420)
0925 09:16:47.549824 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:47.549846 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:47.549984 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:47.550932 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:47.550938 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:47.551471 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:47.551691 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:47.551836 (+ 145us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.553570 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.553576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:47.554996 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:47.555014 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.556346 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.556353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:47.556582 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.557628 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:47.557654 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:47.567313 (+ 9659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:47.583862 (+ 16549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:47.583871 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:47.583876 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:47.585028 (+ 1152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:47.586060 (+ 1032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:47.586095 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:47.586099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:47.586448 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:47.587909 (+ 1461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:47.587915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:47.588249 (+ 334us) client_negotiation.cc:769] Sending connection context
0925 09:16:47.588547 (+ 298us) client_negotiation.cc:240] Negotiation successful
0925 09:16:47.589182 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260925 09:16:47.593212 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:47.550248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.3.19.124:39149)
0925 09:16:47.550419 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:47.550425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:47.550444 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:16:47.550509 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:47.550514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:47.550608 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:47.550720 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:47.552129 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.553358 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.555235 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:47.555962 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:47.556082 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:47.567600 (+ 11518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:47.567608 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:47.567613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:47.567660 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:47.583498 (+ 15838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.585464 (+ 1966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.585471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.585475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.585809 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:47.586750 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:47.586790 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:47.586794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:47.587681 (+ 887us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:47.589808 (+ 2127us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:47.590482 (+ 674us) server_negotiation.cc:299] Negotiation successful
0925 09:16:47.592179 (+ 1697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 09:16:47.601558 16830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685: Bootstrap starting.
I20260925 09:16:47.606084 16830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:47.607510 16830 log.cc:824] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:47.622457 16830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685: No bootstrap required, opened a new log
I20260925 09:16:47.623270 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bfc3ff243e34aa69029f9782ee16910. Latest consensus state: current_term: 1 leader_uuid: "3bfc3ff243e34aa69029f9782ee16910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } } }
I20260925 09:16:47.623667 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:16:47.627252 16946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bfc3ff243e34aa69029f9782ee16910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } } }
I20260925 09:16:47.627562 16946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:47.627956 16948 catalog_manager.cc:1347] Generated new cluster ID: 1eb26451810442e3a5a1c99d34ef2df0
I20260925 09:16:47.628055 16948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:16:47.630967 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bfc3ff243e34aa69029f9782ee16910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } } }
I20260925 09:16:47.631626 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:16:47.649830 16948 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:16:47.651777 16948 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:16:47.670182 16830 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } }
I20260925 09:16:47.671473 16830 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:47.671743 16830 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ae1ed863cd4b7894b39a889e75e685, State: Initialized, Role: FOLLOWER
I20260925 09:16:47.673419 16830 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } }
I20260925 09:16:47.676358 16948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910: Generated new TSK 0
I20260925 09:16:47.677721 16948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:16:47.679670 16955 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } } }
I20260925 09:16:47.680197 16955 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:16:47.683475 16804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bfc3ff243e34aa69029f9782ee16910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ae1ed863cd4b7894b39a889e75e685"
I20260925 09:16:47.683974 16804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:47.687456 16804 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bfc3ff243e34aa69029f9782ee16910 in term 1.
I20260925 09:16:47.687920 16805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bfc3ff243e34aa69029f9782ee16910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ae1ed863cd4b7894b39a889e75e685" is_pre_election: true
I20260925 09:16:47.688279 16805 raft_consensus.cc:2362] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3bfc3ff243e34aa69029f9782ee16910 in term 1. Re-sending same reply.
I20260925 09:16:47.689110 16803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [term 1 FOLLOWER]: Refusing update from remote peer 3bfc3ff243e34aa69029f9782ee16910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:16:47.690441 16918 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:16:47.696518 16830 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:16:47.740504 16955 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bfc3ff243e34aa69029f9782ee16910. Latest consensus state: current_term: 1 leader_uuid: "3bfc3ff243e34aa69029f9782ee16910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } } }
I20260925 09:16:47.740861 16955 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:16:47.740584 16962 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bfc3ff243e34aa69029f9782ee16910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } } }
I20260925 09:16:47.741250 16962 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:16:47.781821 16975 catalog_manager.cc:1259] Loaded cluster ID: 1eb26451810442e3a5a1c99d34ef2df0
I20260925 09:16:47.781934 16975 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685: loading cluster ID for follower catalog manager: success
I20260925 09:16:47.786087 16975 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685: acquiring CA information for follower catalog manager: success
I20260925 09:16:47.788509 16975 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f1ae1ed863cd4b7894b39a889e75e685: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 09:16:48.165576 16918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bfc3ff243e34aa69029f9782ee16910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3736482672495192ab4edcb36a50cd" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46817 } } peers { permanent_uuid: "f1ae1ed863cd4b7894b39a889e75e685" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 41915 } } peers { permanent_uuid: "3bfc3ff243e34aa69029f9782ee16910" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 39149 } } }
I20260925 09:16:48.165966 16918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfc3ff243e34aa69029f9782ee16910 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:48.496295 16944 catalog_manager.cc:1259] Loaded cluster ID: 1eb26451810442e3a5a1c99d34ef2df0
I20260925 09:16:48.496428 16944 catalog_manager.cc:1552] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd: loading cluster ID for follower catalog manager: success
I20260925 09:16:48.499001 16944 catalog_manager.cc:1574] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd: acquiring CA information for follower catalog manager: success
I20260925 09:16:48.500784 16944 catalog_manager.cc:1602] T 00000000000000000000000000000000 P da3736482672495192ab4edcb36a50cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 09:16:48.814908 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 16688
I20260925 09:16:48.816218 16750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:48.928993 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 16688
I20260925 09:16:48.950519 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 16763
I20260925 09:16:48.951591 16825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:49.055874 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 16763
I20260925 09:16:49.076359 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 16837
I20260925 09:16:49.077405 16899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:49.183956 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 16837
2026-09-25T09:16:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4629 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:16:49 dist-test-slave-l6qh krb5kdc[16984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:16:49 dist-test-slave-l6qh krb5kdc[16984](info): set up 2 sockets
Sep 25 09:16:49 dist-test-slave-l6qh krb5kdc[16984](info): commencing operation
krb5kdc: starting...
W20260925 09:16:50.290759 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:16:50 dist-test-slave-l6qh krb5kdc[16984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:16:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:16:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:16:50.537616 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:36577
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:36577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:16:50.861848 16997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:50.862144 16997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:50.862327 16997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:50.869822 16997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:16:50.869930 16997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:50.870036 16997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:50.870141 16997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:16:50.870231 16997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:16:50.879925 16997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:36577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:36577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:50.881198 16997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:50.882530 16997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:50.894644 17002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:50.894685 17003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:50.894718 17005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:50.895438 16997 server_base.cc:1034] running on GCE node
I20260925 09:16:50.896131 16997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:50.897431 16997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:50.898625 16997 hybrid_clock.cc:648] HybridClock initialized: now 1790327810898586 us; error 29 us; skew 500 ppm
I20260925 09:16:50.898916 16997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:50 dist-test-slave-l6qh krb5kdc[16984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327810, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:50.901865 16997 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:16:50.903236 16997 webserver.cc:460] Webserver started at http://127.3.19.126:39651/ using document root <none> and password file <none>
I20260925 09:16:50.903873 16997 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:50.903970 16997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:50.904239 16997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:16:50.907115 16997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "4c9bceba24364858b9effc2d6588b630"
format_stamp: "Formatted at 2026-09-25 09:16:50 on dist-test-slave-l6qh"
server_key: "b63439f809b054a39b3969892c5ee32c"
server_key_iv: "d924458bcb42ea1555a38fbba8d8025d"
server_key_version: "encryptionkey@0"
I20260925 09:16:50.907820 16997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "4c9bceba24364858b9effc2d6588b630"
format_stamp: "Formatted at 2026-09-25 09:16:50 on dist-test-slave-l6qh"
server_key: "b63439f809b054a39b3969892c5ee32c"
server_key_iv: "d924458bcb42ea1555a38fbba8d8025d"
server_key_version: "encryptionkey@0"
I20260925 09:16:50.912958 16997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 09:16:50.916419 17012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:50.917765 16997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 09:16:50.917929 16997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "4c9bceba24364858b9effc2d6588b630"
format_stamp: "Formatted at 2026-09-25 09:16:50 on dist-test-slave-l6qh"
server_key: "b63439f809b054a39b3969892c5ee32c"
server_key_iv: "d924458bcb42ea1555a38fbba8d8025d"
server_key_version: "encryptionkey@0"
I20260925 09:16:50.918123 16997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:50.929643 16997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:50.933494 16997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:50.933774 16997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:50.956652 16997 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:36577
I20260925 09:16:50.956643 17064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:36577 every 8 connection(s)
I20260925 09:16:50.958715 16997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:16:50.959825 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 16997
I20260925 09:16:50.960048 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:16:50.961093 3149 external_mini_cluster.cc:1428] Setting key 9c1e13d2239a7e89b11343a30674c906
I20260925 09:16:50.963523 17065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:50.979251 17065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630: Bootstrap starting.
Sep 25 09:16:50 dist-test-slave-l6qh krb5kdc[16984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:50.983606 17065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:50.985154 17065 log.cc:824] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:50.988595 17065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630: No bootstrap required, opened a new log
I20260925 09:16:50.991794 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:50.963256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.3.19.126:36577)
0925 09:16:50.964068 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:50.964079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:50.964119 (+ 40us) server_negotiation.cc:407] Connection header received
0925 09:16:50.966425 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:50.966447 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:50.967023 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:50.967558 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:50.970197 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:50.972605 (+ 2408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:50.974817 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:50.976029 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:50.976240 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:50.981725 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:50.981747 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:50.981770 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:50.981825 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:50.986507 (+ 4682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:50.987363 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:50.987374 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:50.987384 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:50.987625 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:50.988084 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:50.988092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:50.988096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:50.988861 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:50.989259 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:50.989704 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 09:16:50.990178 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":366,"threads_started":1}
I20260925 09:16:50.996450 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:16:50.997653 17065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9bceba24364858b9effc2d6588b630" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36577 } }
I20260925 09:16:50.998026 17065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:50.998124 17065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9bceba24364858b9effc2d6588b630, State: Initialized, Role: FOLLOWER
I20260925 09:16:50.998701 17065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9bceba24364858b9effc2d6588b630" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36577 } }
I20260925 09:16:50.998934 17065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:16:50.999106 17065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:16:50.999279 17065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:51.002053 17065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9bceba24364858b9effc2d6588b630" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36577 } }
I20260925 09:16:51.002513 17065 leader_election.cc:302] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c9bceba24364858b9effc2d6588b630; no voters:
I20260925 09:16:51.003085 17065 leader_election.cc:288] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:16:51.003206 17070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:16:51.003923 17070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [term 1 LEADER]: Becoming Leader. State: Replica: 4c9bceba24364858b9effc2d6588b630, State: Running, Role: LEADER
I20260925 09:16:51.004395 17070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9bceba24364858b9effc2d6588b630" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36577 } }
I20260925 09:16:51.005070 17065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:16:51.007050 17072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c9bceba24364858b9effc2d6588b630. Latest consensus state: current_term: 1 leader_uuid: "4c9bceba24364858b9effc2d6588b630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9bceba24364858b9effc2d6588b630" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36577 } } }
I20260925 09:16:51.007022 17071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c9bceba24364858b9effc2d6588b630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9bceba24364858b9effc2d6588b630" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 36577 } } }
I20260925 09:16:51.007531 17071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:51.007531 17072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:51.009217 17075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:16:51.014671 17075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:16:51.023461 17075 catalog_manager.cc:1347] Generated new cluster ID: fc91182dfd7e45c5990b847ade7273d4
I20260925 09:16:51.023593 17075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:16:51.031623 17075 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:16:51.032823 17075 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:16:51.043884 17075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c9bceba24364858b9effc2d6588b630: Generated new TSK 0
I20260925 09:16:51.044740 17075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:16:51.058154 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 16997
I20260925 09:16:51.059366 17060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:51.148144 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 16997
2026-09-25T09:16:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1962 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:16:51 dist-test-slave-l6qh krb5kdc[17093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:16:51 dist-test-slave-l6qh krb5kdc[17093](info): set up 2 sockets
Sep 25 09:16:51 dist-test-slave-l6qh krb5kdc[17093](info): commencing operation
krb5kdc: starting...
W20260925 09:16:52.270931 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:16:52 dist-test-slave-l6qh krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:16:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:16:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:16:52.494931 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:38513
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:16:52.816205 17106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:52.816570 17106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:52.816758 17106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:52.824214 17106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:16:52.824359 17106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:52.824481 17106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:52.824587 17106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:16:52.824687 17106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:16:52.834347 17106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:38513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:52.835664 17106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:52.837019 17106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:52.849081 17112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:52.849273 17111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:52.849171 17114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:52.850317 17106 server_base.cc:1034] running on GCE node
I20260925 09:16:52.851135 17106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:52.852566 17106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:52.853792 17106 hybrid_clock.cc:648] HybridClock initialized: now 1790327812853740 us; error 44 us; skew 500 ppm
I20260925 09:16:52.854164 17106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:52 dist-test-slave-l6qh krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327812, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:52.864599 17106 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:16:52.866226 17106 webserver.cc:460] Webserver started at http://127.3.19.126:33129/ using document root <none> and password file <none>
I20260925 09:16:52.866962 17106 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:52.867110 17106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:52.867414 17106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:16:52.870494 17106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "470394a6942e419d98230eb3deeb5703"
format_stamp: "Formatted at 2026-09-25 09:16:52 on dist-test-slave-l6qh"
server_key: "f828e49ea8a502628cbb21532d0730c5"
server_key_iv: "eabbb67e767b544c91345216b91c04e0"
server_key_version: "encryptionkey@0"
I20260925 09:16:52.871264 17106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "470394a6942e419d98230eb3deeb5703"
format_stamp: "Formatted at 2026-09-25 09:16:52 on dist-test-slave-l6qh"
server_key: "f828e49ea8a502628cbb21532d0730c5"
server_key_iv: "eabbb67e767b544c91345216b91c04e0"
server_key_version: "encryptionkey@0"
I20260925 09:16:52.876354 17106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:16:52.879926 17121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:52.881400 17106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:16:52.881606 17106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "470394a6942e419d98230eb3deeb5703"
format_stamp: "Formatted at 2026-09-25 09:16:52 on dist-test-slave-l6qh"
server_key: "f828e49ea8a502628cbb21532d0730c5"
server_key_iv: "eabbb67e767b544c91345216b91c04e0"
server_key_version: "encryptionkey@0"
I20260925 09:16:52.881886 17106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:52.895607 17106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:52.900209 17106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:52.900516 17106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:52.920187 17106 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:38513
I20260925 09:16:52.920223 17172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:38513 every 8 connection(s)
I20260925 09:16:52.921706 17106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:16:52.924724 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 17106
I20260925 09:16:52.924969 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:16:52.925987 3149 external_mini_cluster.cc:1428] Setting key d202ceb4828f2848a6910b79072d1aef
I20260925 09:16:52.926393 17173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:52.942822 17173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Bootstrap starting.
Sep 25 09:16:52 dist-test-slave-l6qh krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:52.946537 17173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:52.947778 17173 log.cc:824] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:52.951081 17173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: No bootstrap required, opened a new log
I20260925 09:16:52.952198 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:52.928506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.3.19.126:38513)
0925 09:16:52.929616 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:52.929633 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:52.930313 (+ 680us) server_negotiation.cc:407] Connection header received
0925 09:16:52.931997 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:52.932020 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:52.932453 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:52.932802 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:52.934842 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:52.936234 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:52.938269 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:52.939036 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:52.939190 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:52.944618 (+ 5428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:52.944627 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:52.944643 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:52.944698 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:52.947958 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:52.948739 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:52.948746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:52.948752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:52.948930 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:52.949388 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:52.949392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:52.949394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:52.949906 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:52.950264 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:52.950635 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 09:16:52.951119 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":424,"threads_started":1}
I20260925 09:16:52.960215 17173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } }
I20260925 09:16:52.960680 17173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:52.960809 17173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 470394a6942e419d98230eb3deeb5703, State: Initialized, Role: FOLLOWER
I20260925 09:16:52.961568 17173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } }
I20260925 09:16:52.961875 17173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:16:52.962052 17173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:16:52.962268 17173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:52.964944 17173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } }
I20260925 09:16:52.965574 17173 leader_election.cc:302] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 470394a6942e419d98230eb3deeb5703; no voters:
I20260925 09:16:52.966288 17173 leader_election.cc:288] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:16:52.966454 17178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:16:52.967972 17178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 1 LEADER]: Becoming Leader. State: Replica: 470394a6942e419d98230eb3deeb5703, State: Running, Role: LEADER
I20260925 09:16:52.968632 17173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:16:52.968510 17178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } }
I20260925 09:16:52.971812 17180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 470394a6942e419d98230eb3deeb5703. Latest consensus state: current_term: 1 leader_uuid: "470394a6942e419d98230eb3deeb5703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } } }
I20260925 09:16:52.971837 17179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "470394a6942e419d98230eb3deeb5703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } } }
I20260925 09:16:52.972290 17180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:52.972290 17179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:52.973178 17186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:16:52.978497 17186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:16:52.996547 17186 catalog_manager.cc:1347] Generated new cluster ID: 4a543d38b1074741a2a3db8701600a92
I20260925 09:16:52.996702 17186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:16:53.016978 17186 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:16:53.018433 17186 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:16:53.028759 17186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Generated new TSK 0
I20260925 09:16:53.029790 17186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:16:53.106395 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:38513
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:16:53.467954 17199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:53.468298 17199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:53.468464 17199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:53.476198 17199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:53.476310 17199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:53.476424 17199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:16:53.486497 17199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:53.487761 17199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:53.489023 17199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:53.502789 17207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:53.502840 17205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:53.503047 17204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:53.504000 17199 server_base.cc:1034] running on GCE node
I20260925 09:16:53.504652 17199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:53.505841 17199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:53.507093 17199 hybrid_clock.cc:648] HybridClock initialized: now 1790327813507040 us; error 57 us; skew 500 ppm
I20260925 09:16:53.507438 17199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:53 dist-test-slave-l6qh krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327813, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:53.510598 17199 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:16:53.512111 17199 webserver.cc:460] Webserver started at http://127.3.19.65:33303/ using document root <none> and password file <none>
I20260925 09:16:53.512814 17199 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:53.512960 17199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:53.513363 17199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:16:53.516445 17199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "0b9dae99899444c0b8a3c91a54d383f9"
format_stamp: "Formatted at 2026-09-25 09:16:53 on dist-test-slave-l6qh"
server_key: "b12358e4d5775253bf99edce05c8aeda"
server_key_iv: "41bb66723f1ad47c649f6a29894da445"
server_key_version: "encryptionkey@0"
I20260925 09:16:53.517221 17199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "0b9dae99899444c0b8a3c91a54d383f9"
format_stamp: "Formatted at 2026-09-25 09:16:53 on dist-test-slave-l6qh"
server_key: "b12358e4d5775253bf99edce05c8aeda"
server_key_iv: "41bb66723f1ad47c649f6a29894da445"
server_key_version: "encryptionkey@0"
I20260925 09:16:53.522619 17199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 09:16:53.526346 17214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:53.527858 17199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:16:53.528056 17199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "0b9dae99899444c0b8a3c91a54d383f9"
format_stamp: "Formatted at 2026-09-25 09:16:53 on dist-test-slave-l6qh"
server_key: "b12358e4d5775253bf99edce05c8aeda"
server_key_iv: "41bb66723f1ad47c649f6a29894da445"
server_key_version: "encryptionkey@0"
I20260925 09:16:53.528286 17199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:53.543689 17199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:53.548290 17199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:53.548641 17199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:53.550009 17199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:16:53.551524 17199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:16:53.551636 17199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:53.551827 17199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:16:53.551909 17199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:53.587074 17199 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:44861
I20260925 09:16:53.587165 17327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:44861 every 8 connection(s)
I20260925 09:16:53.588387 17199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:16:53.595815 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 17199
I20260925 09:16:53.596153 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:16:53.597033 3149 external_mini_cluster.cc:1428] Setting key 9b0972ceff5d787995b3c7e42fe284f0
Sep 25 09:16:53 dist-test-slave-l6qh krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327813, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:53.611438 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:53.591005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:33517 (local address 127.3.19.126:38513)
0925 09:16:53.591469 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:53.591476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:53.592624 (+ 1148us) server_negotiation.cc:407] Connection header received
0925 09:16:53.594581 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:53.594586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:53.594663 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:53.594779 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:53.596393 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:53.597539 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:53.599467 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:53.600106 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:53.600265 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:53.604099 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:53.604108 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:53.604116 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:53.604166 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:53.607920 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:53.608820 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:53.608827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:53.608834 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:53.608997 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:53.609622 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:53.609628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:53.609632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:53.609934 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:53.610047 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:53.610998 (+ 951us) server_negotiation.cc:299] Negotiation successful
0925 09:16:53.611174 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":220,"threads_started":1}
I20260925 09:16:53.613188 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:53.591622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:33517)
0925 09:16:53.592449 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:53.592506 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:53.594276 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:53.594996 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:53.595005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:53.595556 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:53.596180 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:53.596199 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:53.597728 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:53.597735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:53.599306 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:53.599317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:53.600344 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:53.600349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:53.600560 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:53.601349 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:53.601372 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:53.603855 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:53.608111 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:53.608120 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:53.608139 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:53.608672 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:53.609141 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:53.609145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:53.609147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:53.609454 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:53.610067 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:53.610078 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:53.610555 (+ 477us) client_negotiation.cc:769] Sending connection context
0925 09:16:53.610865 (+ 310us) client_negotiation.cc:240] Negotiation successful
0925 09:16:53.611491 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":332,"threads_started":1}
I20260925 09:16:53.614825 17328 heartbeater.cc:347] Connected to a master server at 127.3.19.126:38513
I20260925 09:16:53.615151 17328 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:53.615890 17328 heartbeater.cc:511] Master 127.3.19.126:38513 requested a full tablet report, sending...
I20260925 09:16:53.618115 17138 ts_manager.cc:194] Registered new tserver with Master: 0b9dae99899444c0b8a3c91a54d383f9 (127.3.19.65:44861)
I20260925 09:16:53.619728 17138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:33517
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:16:53.658752 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:38513
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:16:53.996863 17334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:53.997221 17334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:53.997416 17334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:54.004981 17334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:54.005158 17334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:54.005367 17334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:16:54.016340 17334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:54.017830 17334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:54.019183 17334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:54.033310 17340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:54.033576 17342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:54.033958 17339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:54.034175 17334 server_base.cc:1034] running on GCE node
I20260925 09:16:54.034921 17334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:54.035928 17334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:54.037173 17334 hybrid_clock.cc:648] HybridClock initialized: now 1790327814037122 us; error 34 us; skew 500 ppm
I20260925 09:16:54.037611 17334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:54 dist-test-slave-l6qh krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327814, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:54.041442 17334 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:16:54.043226 17334 webserver.cc:460] Webserver started at http://127.3.19.66:40087/ using document root <none> and password file <none>
I20260925 09:16:54.044055 17334 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:54.044215 17334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:54.044577 17334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:16:54.048010 17334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "0f71c11460c0447291b0e4d69de7afb0"
format_stamp: "Formatted at 2026-09-25 09:16:54 on dist-test-slave-l6qh"
server_key: "c55832ba1f3c50fe3ef8096dd7b5e380"
server_key_iv: "9439183cb8e46dfd6661a850339edfa6"
server_key_version: "encryptionkey@0"
I20260925 09:16:54.048859 17334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "0f71c11460c0447291b0e4d69de7afb0"
format_stamp: "Formatted at 2026-09-25 09:16:54 on dist-test-slave-l6qh"
server_key: "c55832ba1f3c50fe3ef8096dd7b5e380"
server_key_iv: "9439183cb8e46dfd6661a850339edfa6"
server_key_version: "encryptionkey@0"
I20260925 09:16:54.055080 17334 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260925 09:16:54.059157 17349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:54.060751 17334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:16:54.060949 17334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "0f71c11460c0447291b0e4d69de7afb0"
format_stamp: "Formatted at 2026-09-25 09:16:54 on dist-test-slave-l6qh"
server_key: "c55832ba1f3c50fe3ef8096dd7b5e380"
server_key_iv: "9439183cb8e46dfd6661a850339edfa6"
server_key_version: "encryptionkey@0"
I20260925 09:16:54.061157 17334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:54.079497 17334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:54.083736 17334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:54.084048 17334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:54.085119 17334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:16:54.086781 17334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:16:54.086854 17334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:54.086942 17334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:16:54.086977 17334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:54.120957 17334 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:38167
I20260925 09:16:54.121002 17462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:38167 every 8 connection(s)
I20260925 09:16:54.122320 17334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:16:54.124218 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 17334
I20260925 09:16:54.124413 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:16:54.124907 3149 external_mini_cluster.cc:1428] Setting key ef72189035167ad414d22347fd9fc9aa
Sep 25 09:16:54 dist-test-slave-l6qh krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327814, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:54.146346 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.125127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:34653 (local address 127.3.19.126:38513)
0925 09:16:54.125655 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:54.125662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:54.126907 (+ 1245us) server_negotiation.cc:407] Connection header received
0925 09:16:54.129802 (+ 2895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:54.129806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:54.129873 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:54.129970 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:54.132178 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.133066 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.135698 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.136147 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.136247 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.140059 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:54.140064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:54.140067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:54.140103 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:54.143048 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:54.143852 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:54.143864 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:54.143867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:54.144002 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:54.144468 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:54.144472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:54.144474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:54.144748 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:54.144908 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:54.145697 (+ 789us) server_negotiation.cc:299] Negotiation successful
0925 09:16:54.145938 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":280,"threads_started":1}
I20260925 09:16:54.147414 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.125782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:34653)
0925 09:16:54.126726 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:54.126777 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:54.129470 (+ 2693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:54.130265 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:54.130280 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:54.131132 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:54.131991 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.132021 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.133270 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.133279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.135525 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.135543 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.136392 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.136401 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.136788 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.137574 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:54.137603 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:54.139804 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:54.143214 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:54.143225 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:54.143244 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:54.143691 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:54.144120 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:54.144125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:54.144129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:54.144333 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:54.144880 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:54.144888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:54.145321 (+ 433us) client_negotiation.cc:769] Sending connection context
0925 09:16:54.145605 (+ 284us) client_negotiation.cc:240] Negotiation successful
0925 09:16:54.146185 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":372,"threads_started":1}
I20260925 09:16:54.148998 17463 heartbeater.cc:347] Connected to a master server at 127.3.19.126:38513
I20260925 09:16:54.149372 17463 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:54.150127 17463 heartbeater.cc:511] Master 127.3.19.126:38513 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
I20260925 09:16:54.151701 17138 ts_manager.cc:194] Registered new tserver with Master: 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167)
I20260925 09:16:54.152697 17138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:34653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:16:54.184849 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:38513
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:16:54.522904 17469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:54.523177 17469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:54.523345 17469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:54.530720 17469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:54.530849 17469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:54.531034 17469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:16:54.541126 17469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:54.542514 17469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:54.543926 17469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:54.557659 17474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:54.557613 17475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:54.557694 17477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:54.558663 17469 server_base.cc:1034] running on GCE node
I20260925 09:16:54.559332 17469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:54.560359 17469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:54.561591 17469 hybrid_clock.cc:648] HybridClock initialized: now 1790327814561525 us; error 58 us; skew 500 ppm
I20260925 09:16:54.561947 17469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:54 dist-test-slave-l6qh krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327814, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:54.565289 17469 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:16:54.566933 17469 webserver.cc:460] Webserver started at http://127.3.19.67:45257/ using document root <none> and password file <none>
I20260925 09:16:54.567699 17469 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:54.567850 17469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:54.568187 17469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:16:54.571578 17469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "e88e973884ba4792941c82bf9a32c7d2"
format_stamp: "Formatted at 2026-09-25 09:16:54 on dist-test-slave-l6qh"
server_key: "66da70c1999f9abb57b38cad06798c4b"
server_key_iv: "0fdba63c966fdc483d14923d99f2bd9b"
server_key_version: "encryptionkey@0"
I20260925 09:16:54.572343 17469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "e88e973884ba4792941c82bf9a32c7d2"
format_stamp: "Formatted at 2026-09-25 09:16:54 on dist-test-slave-l6qh"
server_key: "66da70c1999f9abb57b38cad06798c4b"
server_key_iv: "0fdba63c966fdc483d14923d99f2bd9b"
server_key_version: "encryptionkey@0"
I20260925 09:16:54.578132 17469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 09:16:54.582008 17484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:54.583472 17469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:16:54.583665 17469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "e88e973884ba4792941c82bf9a32c7d2"
format_stamp: "Formatted at 2026-09-25 09:16:54 on dist-test-slave-l6qh"
server_key: "66da70c1999f9abb57b38cad06798c4b"
server_key_iv: "0fdba63c966fdc483d14923d99f2bd9b"
server_key_version: "encryptionkey@0"
I20260925 09:16:54.583887 17469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:54.607241 17469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:54.612078 17469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:54.612433 17469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:54.613718 17469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:16:54.615242 17469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:16:54.615366 17469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:54.615531 17469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:16:54.615613 17469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:54.622727 17328 heartbeater.cc:503] Master 127.3.19.126:38513 was elected leader, sending a full tablet report...
I20260925 09:16:54.653437 17469 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:45087
I20260925 09:16:54.653539 17596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:45087 every 8 connection(s)
I20260925 09:16:54.654910 17469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:16:54.660207 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 17469
I20260925 09:16:54.660408 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:16:54.660979 3149 external_mini_cluster.cc:1428] Setting key 4cf05aebb3b5b0917d99a6872c53a661
Sep 25 09:16:54 dist-test-slave-l6qh krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327814, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:54.677423 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.657745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:35359 (local address 127.3.19.126:38513)
0925 09:16:54.658191 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:54.658196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:54.659296 (+ 1100us) server_negotiation.cc:407] Connection header received
0925 09:16:54.661975 (+ 2679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:54.661980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:54.662057 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:54.662220 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:54.664198 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.665312 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.667029 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.667417 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.667519 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.671192 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:54.671197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:54.671200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:54.671235 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:54.673866 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:54.674610 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:54.674615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:54.674618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:54.674764 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:54.675422 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:54.675429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:54.675435 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:54.675861 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:54.676018 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:54.676844 (+ 826us) server_negotiation.cc:299] Negotiation successful
0925 09:16:54.677078 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":221,"threads_started":1}
I20260925 09:16:54.678402 17600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.658357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:35359)
0925 09:16:54.659099 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:54.659147 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:54.661628 (+ 2481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:54.662516 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:54.662530 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:54.663248 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:54.663924 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.663950 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.665481 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.665485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.666855 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.666866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.667622 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.667628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.667838 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.668505 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:54.668541 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:54.670954 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:54.674023 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:54.674030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:54.674045 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:54.674491 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:54.674929 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:54.674934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:54.674937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:54.675196 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:54.676029 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:54.676037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:54.676445 (+ 408us) client_negotiation.cc:769] Sending connection context
0925 09:16:54.676738 (+ 293us) client_negotiation.cc:240] Negotiation successful
0925 09:16:54.677222 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":284,"threads_started":1}
I20260925 09:16:54.679766 17597 heartbeater.cc:347] Connected to a master server at 127.3.19.126:38513
I20260925 09:16:54.680044 17597 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:54.680706 17597 heartbeater.cc:511] Master 127.3.19.126:38513 requested a full tablet report, sending...
I20260925 09:16:54.682159 17138 ts_manager.cc:194] Registered new tserver with Master: e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087)
I20260925 09:16:54.682984 17138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:35359
I20260925 09:16:54.689517 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:16:54.707355 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.694184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.3.19.126:38513)
0925 09:16:54.694369 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:54.694373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:54.694777 (+ 404us) server_negotiation.cc:407] Connection header received
0925 09:16:54.694959 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:54.694964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:54.695034 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:54.695123 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:54.696527 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.697378 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.698755 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.699400 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.699509 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.702032 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:54.702038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:54.702042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:54.702082 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:54.704956 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:54.705704 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:54.705708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:54.705710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:54.705839 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:54.706162 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:54.706166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:54.706169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:54.706497 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:54.706719 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:54.706955 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 09:16:54.707131 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 09:16:54.711776 17138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:16:54.718784 17138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:16:54.744503 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.731183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:38167 (local address 127.0.0.1:43416)
0925 09:16:54.732474 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:54.732513 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:54.732695 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:54.733563 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:54.733573 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:54.733599 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:54.734088 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.734106 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.736645 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.736654 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.740236 (+ 3582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.740260 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.743350 (+ 3090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.743358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.743652 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.743731 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 09:16:54.743891 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 09:16:54.744107 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"thread_start_us":851,"threads_started":1}
I20260925 09:16:54.744812 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.731422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.3.19.66:38167)
0925 09:16:54.732021 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:54.732032 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:54.732752 (+ 720us) server_negotiation.cc:407] Connection header received
0925 09:16:54.732981 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:54.732991 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:54.733298 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:54.733489 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:54.734323 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.736412 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.740493 (+ 4081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.742669 (+ 2176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.742854 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.743140 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:54.744302 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0925 09:16:54.744488 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":311,"threads_started":1}
I20260925 09:16:54.749399 17397 tablet_service.cc:1467] Processing CreateTablet for tablet f938bd5e66d34710b5ef0a40b973c6d3 (DEFAULT_TABLE table=test-table [id=a0b17bc0c23e4ddaada70dddaaac4828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:16:54.751111 17397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f938bd5e66d34710b5ef0a40b973c6d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:54.752780 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.731966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:45087 (local address 127.0.0.1:38140)
0925 09:16:54.733273 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:54.733291 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:54.733405 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:54.746373 (+ 12968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:54.746378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:54.746407 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:54.746757 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.746770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.748430 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.748437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.750247 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.750266 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.751963 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.751969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.752130 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.752164 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 09:16:54.752271 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 09:16:54.752410 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"thread_start_us":960,"threads_started":1}
I20260925 09:16:54.752854 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.732467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.3.19.67:45087)
0925 09:16:54.745728 (+ 13261us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:54.745736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:54.745770 (+ 34us) server_negotiation.cc:407] Connection header received
0925 09:16:54.745866 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:54.745874 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:54.746145 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:54.746391 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:54.746981 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.748293 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.750409 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.751751 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.751889 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.752116 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:54.752342 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 09:16:54.752509 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13074,"thread_start_us":12945,"threads_started":1}
I20260925 09:16:54.754294 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.732980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34522)
0925 09:16:54.743818 (+ 10838us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:54.743840 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:54.743958 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:54.747021 (+ 3063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:54.747026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:54.747054 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:54.747475 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.747487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.749401 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.749406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.751533 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.751553 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.753473 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.753480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.753653 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.753692 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 09:16:54.753811 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 09:16:54.753942 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10700,"thread_start_us":11059,"threads_started":1}
I20260925 09:16:54.755779 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.733563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.3.19.65:44861)
0925 09:16:54.746414 (+ 12851us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:54.746424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:54.746456 (+ 32us) server_negotiation.cc:407] Connection header received
0925 09:16:54.746572 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:54.746581 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:54.746841 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:54.747042 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:54.747688 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.749207 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.751749 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.753207 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.753404 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.753640 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:54.753855 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 09:16:54.754015 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12666,"thread_start_us":13038,"threads_started":1}
I20260925 09:16:54.757121 17532 tablet_service.cc:1467] Processing CreateTablet for tablet f938bd5e66d34710b5ef0a40b973c6d3 (DEFAULT_TABLE table=test-table [id=a0b17bc0c23e4ddaada70dddaaac4828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:16:54.758700 17532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f938bd5e66d34710b5ef0a40b973c6d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:54.759775 17262 tablet_service.cc:1467] Processing CreateTablet for tablet f938bd5e66d34710b5ef0a40b973c6d3 (DEFAULT_TABLE table=test-table [id=a0b17bc0c23e4ddaada70dddaaac4828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:16:54.761300 17262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f938bd5e66d34710b5ef0a40b973c6d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:54.764937 17616 tablet_bootstrap.cc:492] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0: Bootstrap starting.
I20260925 09:16:54.768164 17616 tablet_bootstrap.cc:654] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:54.769438 17616 log.cc:824] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:54.774507 17618 tablet_bootstrap.cc:492] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2: Bootstrap starting.
I20260925 09:16:54.777429 17619 tablet_bootstrap.cc:492] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Bootstrap starting.
I20260925 09:16:54.778215 17616 tablet_bootstrap.cc:492] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0: No bootstrap required, opened a new log
I20260925 09:16:54.778476 17618 tablet_bootstrap.cc:654] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:54.778522 17616 ts_tablet_manager.cc:1397] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260925 09:16:54.779798 17618 log.cc:824] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:54.780273 17619 tablet_bootstrap.cc:654] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:54.781592 17619 log.cc:824] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:54.783100 17618 tablet_bootstrap.cc:492] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2: No bootstrap required, opened a new log
I20260925 09:16:54.783699 17618 ts_tablet_manager.cc:1397] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 09:16:54.784408 17619 tablet_bootstrap.cc:492] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: No bootstrap required, opened a new log
I20260925 09:16:54.784711 17619 ts_tablet_manager.cc:1397] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 09:16:54.789101 17616 raft_consensus.cc:348] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } }
I20260925 09:16:54.789634 17616 raft_consensus.cc:374] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:54.789822 17616 raft_consensus.cc:729] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f71c11460c0447291b0e4d69de7afb0, State: Initialized, Role: FOLLOWER
I20260925 09:16:54.790573 17616 consensus_queue.cc:260] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } }
I20260925 09:16:54.791898 17463 heartbeater.cc:503] Master 127.3.19.126:38513 was elected leader, sending a full tablet report...
I20260925 09:16:54.792572 17616 ts_tablet_manager.cc:1428] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 09:16:54.794602 17619 raft_consensus.cc:348] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } }
I20260925 09:16:54.795071 17619 raft_consensus.cc:374] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:54.795184 17619 raft_consensus.cc:729] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9dae99899444c0b8a3c91a54d383f9, State: Initialized, Role: FOLLOWER
I20260925 09:16:54.795136 17618 raft_consensus.cc:348] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } }
I20260925 09:16:54.795607 17618 raft_consensus.cc:374] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:54.795751 17618 raft_consensus.cc:729] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e88e973884ba4792941c82bf9a32c7d2, State: Initialized, Role: FOLLOWER
I20260925 09:16:54.795833 17619 consensus_queue.cc:260] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } }
I20260925 09:16:54.796430 17618 consensus_queue.cc:260] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } }
I20260925 09:16:54.800765 17597 heartbeater.cc:503] Master 127.3.19.126:38513 was elected leader, sending a full tablet report...
I20260925 09:16:54.802176 17618 ts_tablet_manager.cc:1428] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260925 09:16:54.803004 17619 ts_tablet_manager.cc:1428] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
I20260925 09:16:54.807703 17622 raft_consensus.cc:482] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:16:54.807925 17622 raft_consensus.cc:504] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } }
I20260925 09:16:54.809556 17622 leader_election.cc:288] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b9dae99899444c0b8a3c91a54d383f9 (127.3.19.65:44861), e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087)
I20260925 09:16:54.817301 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.810235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:53811 (local address 127.3.19.67:45087)
0925 09:16:54.810477 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:54.810486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:54.811233 (+ 747us) server_negotiation.cc:407] Connection header received
0925 09:16:54.811352 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:54.811359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:54.811441 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:54.811527 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:54.812369 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.813477 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.815465 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.816244 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.816344 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.816486 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:54.816876 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 09:16:54.817007 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 09:16:54.817328 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.809806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:46235 (local address 127.3.19.65:44861)
0925 09:16:54.810061 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:54.810071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:54.811022 (+ 951us) server_negotiation.cc:407] Connection header received
0925 09:16:54.811242 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:54.811248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:54.811325 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:54.811447 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:54.812198 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.813434 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.815491 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.816277 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.816452 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.816554 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:54.816912 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 09:16:54.817037 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 09:16:54.817592 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.809693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.3.19.66:46235)
0925 09:16:54.810891 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:54.810917 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:54.811081 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:54.811456 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:54.811459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:54.811478 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:54.812023 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.812042 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.813580 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.813587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.815310 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.815324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.816537 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.816546 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.816740 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.816763 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 09:16:54.816872 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 09:16:54.817004 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":111,"thread_start_us":1156,"threads_started":1}
I20260925 09:16:54.817644 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.810031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:45087 (local address 127.3.19.66:53811)
0925 09:16:54.811135 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:54.811154 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:54.811262 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:54.811557 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:54.811561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:54.811674 (+ 113us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:54.812214 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.812230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.813652 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.813660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.815297 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:54.815312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.816488 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.816495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:54.816713 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.816747 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 09:16:54.816821 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 09:16:54.816972 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":95,"spinlock_wait_cycles":25984,"thread_start_us":459,"threads_started":1}
I20260925 09:16:54.818569 17282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f938bd5e66d34710b5ef0a40b973c6d3" candidate_uuid: "0f71c11460c0447291b0e4d69de7afb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9dae99899444c0b8a3c91a54d383f9" is_pre_election: true
I20260925 09:16:54.818766 17552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f938bd5e66d34710b5ef0a40b973c6d3" candidate_uuid: "0f71c11460c0447291b0e4d69de7afb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e88e973884ba4792941c82bf9a32c7d2" is_pre_election: true
I20260925 09:16:54.819106 17282 raft_consensus.cc:2454] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f71c11460c0447291b0e4d69de7afb0 in term 0.
I20260925 09:16:54.819301 17552 raft_consensus.cc:2454] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f71c11460c0447291b0e4d69de7afb0 in term 0.
I20260925 09:16:54.819821 17352 leader_election.cc:302] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9dae99899444c0b8a3c91a54d383f9, 0f71c11460c0447291b0e4d69de7afb0; no voters:
I20260925 09:16:54.820223 17622 raft_consensus.cc:2781] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:16:54.820382 17622 raft_consensus.cc:482] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:16:54.820499 17622 raft_consensus.cc:3037] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:54.822817 17622 raft_consensus.cc:504] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } }
I20260925 09:16:54.823436 17622 leader_election.cc:288] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [CANDIDATE]: Term 1 election: Requested vote from peers 0b9dae99899444c0b8a3c91a54d383f9 (127.3.19.65:44861), e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087)
I20260925 09:16:54.823909 17282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f938bd5e66d34710b5ef0a40b973c6d3" candidate_uuid: "0f71c11460c0447291b0e4d69de7afb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9dae99899444c0b8a3c91a54d383f9"
I20260925 09:16:54.823966 17552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f938bd5e66d34710b5ef0a40b973c6d3" candidate_uuid: "0f71c11460c0447291b0e4d69de7afb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e88e973884ba4792941c82bf9a32c7d2"
I20260925 09:16:54.824131 17282 raft_consensus.cc:3037] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:54.824164 17552 raft_consensus.cc:3037] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:54.826382 17282 raft_consensus.cc:2454] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f71c11460c0447291b0e4d69de7afb0 in term 1.
I20260925 09:16:54.826388 17552 raft_consensus.cc:2454] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f71c11460c0447291b0e4d69de7afb0 in term 1.
I20260925 09:16:54.826808 17352 leader_election.cc:302] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9dae99899444c0b8a3c91a54d383f9, 0f71c11460c0447291b0e4d69de7afb0; no voters:
I20260925 09:16:54.827113 17622 raft_consensus.cc:2781] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:16:54.827551 17622 raft_consensus.cc:686] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [term 1 LEADER]: Becoming Leader. State: Replica: 0f71c11460c0447291b0e4d69de7afb0, State: Running, Role: LEADER
I20260925 09:16:54.828061 17622 consensus_queue.cc:237] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } }
I20260925 09:16:54.833308 17137 catalog_manager.cc:5495] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66). New cstate: current_term: 1 leader_uuid: "0f71c11460c0447291b0e4d69de7afb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } health_report { overall_health: UNKNOWN } } }
W20260925 09:16:54.840780 17329 tablet.cc:2369] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:16:54.870603 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.864906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.3.19.66:38167)
0925 09:16:54.865166 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:54.865178 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:54.865198 (+ 20us) server_negotiation.cc:407] Connection header received
0925 09:16:54.865323 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:54.865327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:54.865411 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:54.865541 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:16:54.865885 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.866692 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.868154 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.868600 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.868736 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.869290 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:16:54.869972 (+ 682us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:16:54.870133 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:54.870322 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 09:16:54.870392 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 09:16:54.874287 17464 tablet.cc:2369] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:16:54.879000 17282 raft_consensus.cc:1261] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Refusing update from remote peer 0f71c11460c0447291b0e4d69de7afb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:16:54.879000 17552 raft_consensus.cc:1261] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2 [term 1 FOLLOWER]: Refusing update from remote peer 0f71c11460c0447291b0e4d69de7afb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:16:54.880048 17627 consensus_queue.cc:1035] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:16:54.880405 17622 consensus_queue.cc:1035] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:16:54.895687 17631 mvcc.cc:205] Tried to move back new op lower bound from 7333182729734524928 to 7333182729539792896. Current Snapshot: MvccSnapshot[applied={T|T < 7333182729734524928}]
I20260925 09:16:54.909054 17138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:16:54.909405 17138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:16:54.913126 17138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Sending DeleteTablet for 3 replicas of tablet f938bd5e66d34710b5ef0a40b973c6d3
I20260925 09:16:54.914435 17532 tablet_service.cc:1514] Processing DeleteTablet for tablet f938bd5e66d34710b5ef0a40b973c6d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:16:54 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:38140
I20260925 09:16:54.914515 17262 tablet_service.cc:1514] Processing DeleteTablet for tablet f938bd5e66d34710b5ef0a40b973c6d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:16:54 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:34522
I20260925 09:16:54.914592 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 17199
I20260925 09:16:54.915587 17397 tablet_service.cc:1514] Processing DeleteTablet for tablet f938bd5e66d34710b5ef0a40b973c6d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:16:54 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:43416
Sep 25 09:16:54 dist-test-slave-l6qh krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:16:54.939849 17643 tablet_replica.cc:330] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0: stopping tablet replica
I20260925 09:16:54.940821 17643 raft_consensus.cc:2229] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:16:54.941866 17643 raft_consensus.cc:2258] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:16:54.945290 17643 ts_tablet_manager.cc:1905] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:16:54.946195 17640 tablet_replica.cc:330] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: stopping tablet replica
I20260925 09:16:54.946941 17640 raft_consensus.cc:2229] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:16:54.947736 17640 raft_consensus.cc:2258] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:16:54.948925 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:54.915790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.3.19.65:44861)
0925 09:16:54.915983 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:54.915991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:54.928140 (+ 12149us) server_negotiation.cc:407] Connection header received
0925 09:16:54.928626 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:54.928633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:54.928734 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:54.928815 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:54.930882 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.932125 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.934261 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:54.935022 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:54.935127 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:54.941201 (+ 6074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:54.941245 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:54.941266 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:54.941320 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:54.945293 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:54.946258 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:54.946268 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:54.946272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:54.946468 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:54.946974 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:54.946979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:54.946983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:54.947439 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:54.947722 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:54.948302 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 09:16:54.948548 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 09:16:54.950397 17323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:54.951023 17641 tablet_replica.cc:330] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2: stopping tablet replica
I20260925 09:16:54.953694 17641 raft_consensus.cc:2229] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:16:54.954312 17641 raft_consensus.cc:2258] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:16:54.956666 17641 ts_tablet_manager.cc:1905] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:16:54.965927 17643 ts_tablet_manager.cc:1918] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:16:54.968945 17643 log.cc:1196] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/f938bd5e66d34710b5ef0a40b973c6d3
I20260925 09:16:54.969717 17641 ts_tablet_manager.cc:1918] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:16:54.969998 17641 log.cc:1196] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/f938bd5e66d34710b5ef0a40b973c6d3
I20260925 09:16:54.969980 17643 ts_tablet_manager.cc:1939] T f938bd5e66d34710b5ef0a40b973c6d3 P 0f71c11460c0447291b0e4d69de7afb0: Deleting consensus metadata
I20260925 09:16:54.970623 17641 ts_tablet_manager.cc:1939] T f938bd5e66d34710b5ef0a40b973c6d3 P e88e973884ba4792941c82bf9a32c7d2: Deleting consensus metadata
I20260925 09:16:54.972508 17125 catalog_manager.cc:4841] TS e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087): tablet f938bd5e66d34710b5ef0a40b973c6d3 (table test-table [id=a0b17bc0c23e4ddaada70dddaaac4828]) successfully deleted
I20260925 09:16:54.972819 17124 catalog_manager.cc:4841] TS 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167): tablet f938bd5e66d34710b5ef0a40b973c6d3 (table test-table [id=a0b17bc0c23e4ddaada70dddaaac4828]) successfully deleted
I20260925 09:16:55.062266 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 17199
W20260925 09:16:55.080768 17124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.19.65:44861 (error 108)
I20260925 09:16:55.080911 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 17334
I20260925 09:16:55.082878 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:55.082400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34544)
0925 09:16:55.082593 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:55.082713 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 09:16:55.083225 17124 catalog_manager.cc:4584] TS 0b9dae99899444c0b8a3c91a54d383f9 (127.3.19.65:44861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f938bd5e66d34710b5ef0a40b973c6d3: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Sep 25 09:16:55 dist-test-slave-l6qh krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:16:55.099675 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:55.082113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.3.19.66:38167)
0925 09:16:55.082299 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:55.082305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:55.082326 (+ 21us) server_negotiation.cc:407] Connection header received
0925 09:16:55.082392 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:55.082398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:55.082531 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:55.082638 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:55.084372 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:55.085371 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:55.087212 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:55.087637 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:55.087738 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:55.093191 (+ 5453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:55.093206 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:55.093222 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:55.093326 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:55.096433 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:55.097219 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:55.097478 (+ 259us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:55.097480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:55.097635 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:55.098042 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:55.098048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:55.098051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:55.098444 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:55.098787 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:55.099046 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 09:16:55.099271 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 09:16:55.100425 17458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:55.137089 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:55.136479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34548)
0925 09:16:55.136741 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:55.136869 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 09:16:55.138279 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:55.137875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34554)
0925 09:16:55.138038 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:55.138125 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 09:16:55.199052 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:55.198464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34570)
0925 09:16:55.198726 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:55.198842 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 09:16:55.200165 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:55.199765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34580)
0925 09:16:55.199942 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:55.200024 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 09:16:55.215978 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 17334
I20260925 09:16:55.236119 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 17469
Sep 25 09:16:55 dist-test-slave-l6qh krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:16:55.252115 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:55.236845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.3.19.67:45087)
0925 09:16:55.237011 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:55.237016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:55.237034 (+ 18us) server_negotiation.cc:407] Connection header received
0925 09:16:55.237149 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:55.237153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:55.237215 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:55.237393 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:55.239077 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:55.239834 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:55.241317 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:55.241707 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:55.241856 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:55.246440 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:55.246455 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:55.246471 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:55.246510 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:55.249145 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:55.250039 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:55.250050 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:55.250058 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:55.250192 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:55.250591 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:55.250597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:55.250601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:55.250964 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:55.251263 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:55.251554 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 09:16:55.251777 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 09:16:55.252714 17592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:55.281601 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:55.281003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34582)
0925 09:16:55.281280 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:55.281410 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 09:16:55.282913 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:55.282429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34598)
0925 09:16:55.282657 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:55.282761 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260925 09:16:55.342388 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 17469
I20260925 09:16:55.359112 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 17106
I20260925 09:16:55.360263 17168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:55.471599 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:55.470718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34604)
0925 09:16:55.470983 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:55.471142 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 09:16:55.471787 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 17106
I20260925 09:16:55.494791 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:38513
--webserver_interface=127.3.19.126
--webserver_port=33129
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:16:55.849051 17649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:55.849386 17649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:55.849480 17649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:55.856819 17649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:16:55.856945 17649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:55.856993 17649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:55.857043 17649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:16:55.857090 17649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:16:55.867491 17649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:38513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=33129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:55.868963 17649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:55.870355 17649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:55.895797 17658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:55.895788 17655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:55.895833 17656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:55.896515 17649 server_base.cc:1034] running on GCE node
I20260925 09:16:55.897719 17649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:55.899531 17649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:55.900811 17649 hybrid_clock.cc:648] HybridClock initialized: now 1790327815900754 us; error 74 us; skew 500 ppm
I20260925 09:16:55.901319 17649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:55 dist-test-slave-l6qh krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:55.905272 17649 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:16:55.907162 17649 webserver.cc:460] Webserver started at http://127.3.19.126:33129/ using document root <none> and password file <none>
I20260925 09:16:55.907997 17649 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:55.908183 17649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:55.914757 17649 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 09:16:55.918110 17665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:55.919790 17649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:16:55.920010 17649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "470394a6942e419d98230eb3deeb5703"
format_stamp: "Formatted at 2026-09-25 09:16:52 on dist-test-slave-l6qh"
server_key: "f828e49ea8a502628cbb21532d0730c5"
server_key_iv: "eabbb67e767b544c91345216b91c04e0"
server_key_version: "encryptionkey@0"
I20260925 09:16:55.921212 17649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:55.934860 17649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:55.939463 17649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:55.939772 17649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:55.960162 17649 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:38513
I20260925 09:16:55.960186 17716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:38513 every 8 connection(s)
I20260925 09:16:55.961851 17649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:16:55.969197 17717 sys_catalog.cc:263] Verifying existing consensus state
I20260925 09:16:55.970492 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 17649
I20260925 09:16:55.971510 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:44861
--local_ip_for_outbound_sockets=127.3.19.65
--tserver_master_addrs=127.3.19.126:38513
--webserver_port=33303
--webserver_interface=127.3.19.65
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:16:55.974537 17717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Bootstrap starting.
I20260925 09:16:55.997035 17717 log.cc:824] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:56.009987 17717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 09:16:56.010589 17717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Bootstrap complete.
I20260925 09:16:56.020020 17717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } }
I20260925 09:16:56.020922 17717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 470394a6942e419d98230eb3deeb5703, State: Initialized, Role: FOLLOWER
I20260925 09:16:56.021638 17717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } }
I20260925 09:16:56.021905 17717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:16:56.022025 17717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:16:56.022225 17717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 1 FOLLOWER]: Advancing to term 2
I20260925 09:16:56.026566 17717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } }
I20260925 09:16:56.027158 17717 leader_election.cc:302] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 470394a6942e419d98230eb3deeb5703; no voters:
I20260925 09:16:56.027837 17717 leader_election.cc:288] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 09:16:56.027985 17721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 09:16:56.030594 17717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:16:56.035213 17721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 2 LEADER]: Becoming Leader. State: Replica: 470394a6942e419d98230eb3deeb5703, State: Running, Role: LEADER
I20260925 09:16:56.035905 17721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } }
I20260925 09:16:56.038942 17722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "470394a6942e419d98230eb3deeb5703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } } }
I20260925 09:16:56.039479 17722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:56.040148 17723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 470394a6942e419d98230eb3deeb5703. Latest consensus state: current_term: 2 leader_uuid: "470394a6942e419d98230eb3deeb5703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } } }
I20260925 09:16:56.040431 17723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:56.040485 17732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:16:56.063271 17732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:16:56.065297 17732 catalog_manager.cc:1259] Loaded cluster ID: 4a543d38b1074741a2a3db8701600a92
I20260925 09:16:56.065543 17732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:16:56.069942 17732 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 09:16:56.094450 17732 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 09:16:56.104008 17732 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:16:56.105926 17732 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Loaded TSK: 0
I20260925 09:16:56.107616 17732 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 09:16:56.340662 17719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:56.341024 17719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:56.341199 17719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:56.348866 17719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:56.349052 17719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:56.349277 17719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:16:56.359619 17719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:44861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=33303
--tserver_master_addrs=127.3.19.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:56.361016 17719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:56.362414 17719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:56.377789 17745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:56.377832 17746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:56.379420 17748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:56.380349 17719 server_base.cc:1034] running on GCE node
I20260925 09:16:56.381076 17719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:56.382268 17719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:56.383502 17719 hybrid_clock.cc:648] HybridClock initialized: now 1790327816383458 us; error 44 us; skew 500 ppm
I20260925 09:16:56.383854 17719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:56 dist-test-slave-l6qh krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327816, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:56.387333 17719 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:16:56.389207 17719 webserver.cc:460] Webserver started at http://127.3.19.65:33303/ using document root <none> and password file <none>
I20260925 09:16:56.390219 17719 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:56.390410 17719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:56.396701 17719 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 09:16:56.400034 17755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:56.401633 17719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:16:56.401819 17719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "0b9dae99899444c0b8a3c91a54d383f9"
format_stamp: "Formatted at 2026-09-25 09:16:53 on dist-test-slave-l6qh"
server_key: "b12358e4d5775253bf99edce05c8aeda"
server_key_iv: "41bb66723f1ad47c649f6a29894da445"
server_key_version: "encryptionkey@0"
I20260925 09:16:56.402859 17719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:56.421883 17719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:56.426435 17719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:56.426728 17719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:56.427798 17719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:16:56.429828 17762 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 09:16:56.435724 17719 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 09:16:56.435887 17719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 09:16:56.436030 17719 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 09:16:56.439481 17719 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 09:16:56.439594 17719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.001s
I20260925 09:16:56.439996 17762 tablet_bootstrap.cc:492] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Bootstrap starting.
I20260925 09:16:56.449347 17762 log.cc:824] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:56.464627 17762 tablet_bootstrap.cc:492] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 09:16:56.465705 17762 tablet_bootstrap.cc:492] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Bootstrap complete.
I20260925 09:16:56.466375 17762 ts_tablet_manager.cc:1397] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.004s
I20260925 09:16:56.478016 17762 raft_consensus.cc:348] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } }
I20260925 09:16:56.479883 17762 raft_consensus.cc:729] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9dae99899444c0b8a3c91a54d383f9, State: Initialized, Role: FOLLOWER
I20260925 09:16:56.480768 17762 consensus_queue.cc:260] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } }
I20260925 09:16:56.481930 17719 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:44861
I20260925 09:16:56.481968 17870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:44861 every 8 connection(s)
I20260925 09:16:56.482844 17762 ts_tablet_manager.cc:1428] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260925 09:16:56.483323 17719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:16:56.490397 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 17719
I20260925 09:16:56.491112 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:38167
--local_ip_for_outbound_sockets=127.3.19.66
--tserver_master_addrs=127.3.19.126:38513
--webserver_port=40087
--webserver_interface=127.3.19.66
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 09:16:56 dist-test-slave-l6qh krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327816, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:56.506917 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:56.486151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:43813)
0925 09:16:56.487079 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:56.487132 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:56.487629 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:56.488928 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:56.488936 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:56.489788 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:56.490445 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:56.490497 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:56.492457 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:56.492463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:56.493844 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:56.493857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:56.494881 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:56.494888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:56.495194 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:56.495912 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:56.495940 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:56.498258 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:56.502160 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:56.502171 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:56.502188 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:56.502713 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:56.503191 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:56.503196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:56.503200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:56.503387 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:56.504189 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:56.504197 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:56.504578 (+ 381us) client_negotiation.cc:769] Sending connection context
0925 09:16:56.504880 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 09:16:56.505495 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":423,"threads_started":1}
I20260925 09:16:56.506912 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:56.486281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:43813 (local address 127.3.19.126:38513)
0925 09:16:56.486866 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:56.486878 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:56.487279 (+ 401us) server_negotiation.cc:407] Connection header received
0925 09:16:56.488110 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:56.488124 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:56.488463 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:56.488768 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:56.490739 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:56.492279 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:56.494003 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:56.494670 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:56.494892 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:56.498565 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:56.498575 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:56.498592 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:56.498633 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:56.501977 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:56.502884 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:56.502892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:56.502898 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:56.503065 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:56.503530 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:56.503534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:56.503536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:56.504044 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:56.504273 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:56.505098 (+ 825us) server_negotiation.cc:299] Negotiation successful
0925 09:16:56.505548 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":259,"threads_started":1}
I20260925 09:16:56.510411 17871 heartbeater.cc:347] Connected to a master server at 127.3.19.126:38513
I20260925 09:16:56.510746 17871 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:56.511474 17871 heartbeater.cc:511] Master 127.3.19.126:38513 requested a full tablet report, sending...
I20260925 09:16:56.515234 17682 ts_manager.cc:194] Registered new tserver with Master: 0b9dae99899444c0b8a3c91a54d383f9 (127.3.19.65:44861)
I20260925 09:16:56.521730 17682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:43813
Sep 25 09:16:56 dist-test-slave-l6qh krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:16:56.536948 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:56.519908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34618)
0925 09:16:56.520541 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:56.520579 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:56.520779 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:56.522070 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:56.522078 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:56.523025 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:56.523346 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:56.523357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:56.524889 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:56.524897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:56.526324 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:56.526337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:56.527203 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:56.527208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:56.527446 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:56.528138 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:56.528157 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:56.530357 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:56.533964 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:56.533977 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:56.533984 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:56.534406 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:56.534816 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:56.534821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:56.534824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:56.534974 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:56.535660 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:56.535668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:56.535867 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 09:16:56.536075 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 09:16:56.536475 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":276,"threads_started":1}
I20260925 09:16:56.536969 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:56.520120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.3.19.65:44861)
0925 09:16:56.521413 (+ 1293us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:56.521422 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:56.521454 (+ 32us) server_negotiation.cc:407] Connection header received
0925 09:16:56.521553 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:56.521561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:56.521844 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:56.522912 (+ 1068us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:56.523522 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:56.524733 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:56.526465 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:56.527035 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:56.527192 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:56.530608 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:56.530640 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:56.530660 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:56.530711 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:56.533807 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:56.534564 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:56.534571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:56.534573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:56.534699 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:56.535087 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:56.535093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:56.535096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:56.535459 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:56.535700 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:56.536260 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 09:16:56.536565 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133,"thread_start_us":936,"threads_started":1}
I20260925 09:16:56.538901 17805 tablet_service.cc:1514] Processing DeleteTablet for tablet f938bd5e66d34710b5ef0a40b973c6d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:16:54 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:34618
I20260925 09:16:56.541064 17878 tablet_replica.cc:330] stopping tablet replica
I20260925 09:16:56.541563 17878 raft_consensus.cc:2229] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:16:56.541965 17878 raft_consensus.cc:2258] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:16:56.543716 17878 ts_tablet_manager.cc:1905] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:16:56.550796 17878 ts_tablet_manager.cc:1918] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:16:56.550947 17878 log.cc:1196] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/f938bd5e66d34710b5ef0a40b973c6d3
I20260925 09:16:56.551424 17878 ts_tablet_manager.cc:1939] T f938bd5e66d34710b5ef0a40b973c6d3 P 0b9dae99899444c0b8a3c91a54d383f9: Deleting consensus metadata
I20260925 09:16:56.553643 17668 catalog_manager.cc:4841] TS 0b9dae99899444c0b8a3c91a54d383f9 (127.3.19.65:44861): tablet f938bd5e66d34710b5ef0a40b973c6d3 (table test-table [id=a0b17bc0c23e4ddaada70dddaaac4828]) successfully deleted
W20260925 09:16:56.829661 17875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:56.830010 17875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:56.830183 17875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:56.838233 17875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:56.838397 17875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:56.838584 17875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:16:56.848433 17875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:38167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=40087
--tserver_master_addrs=127.3.19.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:56.849845 17875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:56.851174 17875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:56.864593 17883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:56.864512 17884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:56.864562 17886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:56.865451 17875 server_base.cc:1034] running on GCE node
I20260925 09:16:56.866061 17875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:56.866982 17875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:56.868192 17875 hybrid_clock.cc:648] HybridClock initialized: now 1790327816868127 us; error 52 us; skew 500 ppm
I20260925 09:16:56.868523 17875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:56 dist-test-slave-l6qh krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327816, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:56.871706 17875 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:16:56.873107 17875 webserver.cc:460] Webserver started at http://127.3.19.66:40087/ using document root <none> and password file <none>
I20260925 09:16:56.873842 17875 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:56.873952 17875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:56.879299 17875 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 09:16:56.882108 17893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:56.883443 17875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 09:16:56.883620 17875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "0f71c11460c0447291b0e4d69de7afb0"
format_stamp: "Formatted at 2026-09-25 09:16:54 on dist-test-slave-l6qh"
server_key: "c55832ba1f3c50fe3ef8096dd7b5e380"
server_key_iv: "9439183cb8e46dfd6661a850339edfa6"
server_key_version: "encryptionkey@0"
I20260925 09:16:56.884668 17875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:56.909982 17875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:56.914155 17875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:56.914430 17875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:56.915444 17875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:16:56.916878 17875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:16:56.917004 17875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:56.917131 17875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:16:56.917222 17875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:56.950474 17875 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:38167
I20260925 09:16:56.950560 18005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:38167 every 8 connection(s)
I20260925 09:16:56.951937 17875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:16:56.957378 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 17875
I20260925 09:16:56.958189 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:45087
--local_ip_for_outbound_sockets=127.3.19.67
--tserver_master_addrs=127.3.19.126:38513
--webserver_port=45257
--webserver_interface=127.3.19.67
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 09:16:56 dist-test-slave-l6qh krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327816, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:56.975450 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:56.954451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:46939 (local address 127.3.19.126:38513)
0925 09:16:56.954674 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:56.954681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:56.955807 (+ 1126us) server_negotiation.cc:407] Connection header received
0925 09:16:56.957976 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:56.957980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:56.958051 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:56.958152 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:56.960374 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:56.961468 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:56.964146 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:56.964833 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:56.964962 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:56.969301 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:56.969307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:56.969310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:56.969346 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:56.972238 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:56.973005 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:56.973013 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:56.973016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:56.973161 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:56.973631 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:56.973638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:56.973641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:56.973923 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:56.974078 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:56.974821 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 09:16:56.975062 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 09:16:56.976557 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:56.954949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:46939)
0925 09:16:56.955643 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:56.955685 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:56.957657 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:56.958437 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:56.958451 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:56.959255 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:56.960162 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:56.960188 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:56.961659 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:56.961667 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:56.963950 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:56.963969 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:56.965093 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:56.965103 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:56.965476 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:56.966344 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:56.966381 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:56.968999 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:56.972401 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:56.972411 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:56.972430 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:56.972876 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:56.973312 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:56.973317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:56.973321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:56.973517 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:56.974065 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:56.974073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:56.974434 (+ 361us) client_negotiation.cc:769] Sending connection context
0925 09:16:56.974707 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 09:16:56.975273 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":282,"threads_started":1}
I20260925 09:16:56.978183 18006 heartbeater.cc:347] Connected to a master server at 127.3.19.126:38513
I20260925 09:16:56.978582 18006 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:56.979271 18006 heartbeater.cc:511] Master 127.3.19.126:38513 requested a full tablet report, sending...
I20260925 09:16:56.980894 17682 ts_manager.cc:194] Registered new tserver with Master: 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167)
I20260925 09:16:56.981873 17682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:46939
W20260925 09:16:57.308279 18009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:57.308578 18009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:57.308736 18009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:57.316031 18009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:57.316151 18009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:57.316344 18009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:16:57.326274 18009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:45087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=45257
--tserver_master_addrs=127.3.19.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:57.327656 18009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:57.328963 18009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:57.343317 18015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:57.343357 18017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:57.343426 18014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:57.349639 18009 server_base.cc:1034] running on GCE node
I20260925 09:16:57.350353 18009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:57.351398 18009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:57.352643 18009 hybrid_clock.cc:648] HybridClock initialized: now 1790327817352567 us; error 66 us; skew 500 ppm
I20260925 09:16:57.352963 18009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:57 dist-test-slave-l6qh krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327817, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:57.356184 18009 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:16:57.357925 18009 webserver.cc:460] Webserver started at http://127.3.19.67:45257/ using document root <none> and password file <none>
I20260925 09:16:57.358691 18009 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:57.358871 18009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:57.364473 18009 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 09:16:57.367405 18024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:57.368782 18009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:16:57.368970 18009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "e88e973884ba4792941c82bf9a32c7d2"
format_stamp: "Formatted at 2026-09-25 09:16:54 on dist-test-slave-l6qh"
server_key: "66da70c1999f9abb57b38cad06798c4b"
server_key_iv: "0fdba63c966fdc483d14923d99f2bd9b"
server_key_version: "encryptionkey@0"
I20260925 09:16:57.370182 18009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:57.417017 18009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:57.421509 18009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:57.421788 18009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:57.422724 18009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:16:57.424057 18009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:16:57.424180 18009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:57.424252 18009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:16:57.424316 18009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:57.458478 18009 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:45087
I20260925 09:16:57.458568 18136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:45087 every 8 connection(s)
I20260925 09:16:57.459997 18009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:16:57.461211 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 18009
Sep 25 09:16:57 dist-test-slave-l6qh krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327817, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:16:57.477058 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.462375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57218 (local address 127.3.19.126:38513)
0925 09:16:57.462629 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:57.462636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:57.463066 (+ 430us) server_negotiation.cc:407] Connection header received
0925 09:16:57.463269 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:57.463283 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:57.463392 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:57.463534 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:57.465075 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.465877 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.467521 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.467960 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.468096 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.471061 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:57.471070 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:57.471075 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:57.471119 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:57.474245 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:57.474928 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:57.474937 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:57.474940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:57.475118 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:57.475547 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:57.475552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:57.475555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:57.475875 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:57.476222 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:57.476466 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 09:16:57.476712 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 09:16:57.483207 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.463056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:47453 (local address 127.3.19.126:38513)
0925 09:16:57.463554 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:57.463559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:57.464614 (+ 1055us) server_negotiation.cc:407] Connection header received
0925 09:16:57.466546 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:57.466552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:57.466638 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:57.466765 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:57.468375 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.469545 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.471117 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.471849 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.471962 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.475904 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:57.475913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:57.475920 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:57.475957 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:57.479774 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:57.480578 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:57.480587 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:57.480590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:57.480728 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:57.481329 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:57.481335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:57.481337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:57.481613 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:57.481788 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:57.482526 (+ 738us) server_negotiation.cc:299] Negotiation successful
0925 09:16:57.482773 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":270,"threads_started":1}
I20260925 09:16:57.484190 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.463555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:47453)
0925 09:16:57.464432 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:57.464477 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:57.466284 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:57.467023 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:57.467033 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:57.467579 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:16:57.468202 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:57.468230 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.469687 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.469690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:57.470988 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:57.471003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.472100 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.472109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:57.472377 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.473010 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:16:57.473044 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:16:57.475686 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:16:57.479937 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:57.479947 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:57.479966 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:57.480439 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:57.480850 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:16:57.480856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:16:57.480861 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:16:57.481151 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:16:57.481761 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:16:57.481769 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:16:57.482153 (+ 384us) client_negotiation.cc:769] Sending connection context
0925 09:16:57.482430 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 09:16:57.482974 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":362,"threads_started":1}
I20260925 09:16:57.486459 18137 heartbeater.cc:347] Connected to a master server at 127.3.19.126:38513
I20260925 09:16:57.486820 18137 heartbeater.cc:464] Registering TS with master...
I20260925 09:16:57.487673 18137 heartbeater.cc:511] Master 127.3.19.126:38513 requested a full tablet report, sending...
I20260925 09:16:57.489560 17682 ts_manager.cc:194] Registered new tserver with Master: e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087)
I20260925 09:16:57.490468 17682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:47453
I20260925 09:16:57.492949 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:16:57.500768 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.494408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57226 (local address 127.3.19.126:38513)
0925 09:16:57.494609 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:57.494617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:57.494859 (+ 242us) server_negotiation.cc:407] Connection header received
0925 09:16:57.495125 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:57.495143 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:57.495276 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:57.495441 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:16:57.495952 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.496709 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.498380 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.498833 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.498984 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.499576 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:16:57.500095 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:16:57.500275 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:57.500363 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 09:16:57.500458 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 09:16:57.501883 17682 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:16:57.509817 17682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:16:57.523108 17805 tablet_service.cc:1467] Processing CreateTablet for tablet 2e744337f7834c9185e7e91fb182d49a (DEFAULT_TABLE table=test-table [id=e551f2c4b80f4805bc5ef1cfc23fb0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:16:57.524039 17805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e744337f7834c9185e7e91fb182d49a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:57.526768 17871 heartbeater.cc:503] Master 127.3.19.126:38513 was elected leader, sending a full tablet report...
I20260925 09:16:57.530779 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.520525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:38167 (local address 127.0.0.1:43446)
0925 09:16:57.521663 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:57.521679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:57.521855 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:57.522871 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:57.522876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:57.522912 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:57.523281 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:57.523294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.525166 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.525172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:57.527553 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:57.527567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.529837 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.529843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:57.530064 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.530137 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 09:16:57.530230 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 09:16:57.530375 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"mutex_wait_us":1,"thread_start_us":919,"threads_started":1}
I20260925 09:16:57.530915 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.520783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.3.19.66:38167)
0925 09:16:57.521412 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:57.521424 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:57.521785 (+ 361us) server_negotiation.cc:407] Connection header received
0925 09:16:57.522117 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:57.522126 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:57.522351 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:57.522576 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:57.523477 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.524981 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.527784 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.529329 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.529464 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.529678 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:57.530353 (+ 675us) server_negotiation.cc:299] Negotiation successful
0925 09:16:57.530552 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":308,"threads_started":1}
I20260925 09:16:57.531880 18151 tablet_bootstrap.cc:492] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9: Bootstrap starting.
I20260925 09:16:57.533985 18151 tablet_bootstrap.cc:654] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:57.534545 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.521756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.3.19.67:45087)
0925 09:16:57.524370 (+ 2614us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:57.524380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:57.524416 (+ 36us) server_negotiation.cc:407] Connection header received
0925 09:16:57.524518 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:57.524526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:57.524814 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:57.525022 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:57.525840 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.527813 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.530255 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.533316 (+ 3061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.533484 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.533747 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:57.534008 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 09:16:57.534170 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2438,"thread_start_us":2283,"threads_started":1}
I20260925 09:16:57.534582 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.521008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:45087 (local address 127.0.0.1:38160)
0925 09:16:57.522200 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:57.522220 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:57.522340 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:57.525018 (+ 2678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:57.525023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:57.525049 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:57.525516 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:57.525528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.528000 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.528007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:57.530074 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:57.530087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.533564 (+ 3477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.533571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:57.533776 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.533818 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 09:16:57.533937 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 09:16:57.534088 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"mutex_wait_us":1,"thread_start_us":1367,"threads_started":1}
I20260925 09:16:57.535969 17941 tablet_service.cc:1467] Processing CreateTablet for tablet 2e744337f7834c9185e7e91fb182d49a (DEFAULT_TABLE table=test-table [id=e551f2c4b80f4805bc5ef1cfc23fb0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:16:57.537647 17941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e744337f7834c9185e7e91fb182d49a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:57.538096 18151 tablet_bootstrap.cc:492] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9: No bootstrap required, opened a new log
I20260925 09:16:57.538316 18151 ts_tablet_manager.cc:1397] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 09:16:57.539104 18151 raft_consensus.cc:348] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:16:57.539368 18151 raft_consensus.cc:374] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:57.539523 18151 raft_consensus.cc:729] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9dae99899444c0b8a3c91a54d383f9, State: Initialized, Role: FOLLOWER
I20260925 09:16:57.539143 18072 tablet_service.cc:1467] Processing CreateTablet for tablet 2e744337f7834c9185e7e91fb182d49a (DEFAULT_TABLE table=test-table [id=e551f2c4b80f4805bc5ef1cfc23fb0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:16:57.539896 18151 consensus_queue.cc:260] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:16:57.540692 18072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e744337f7834c9185e7e91fb182d49a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:16:57.550783 18151 ts_tablet_manager.cc:1428] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260925 09:16:57.554112 18154 tablet_bootstrap.cc:492] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0: Bootstrap starting.
I20260925 09:16:57.556473 18155 tablet_bootstrap.cc:492] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2: Bootstrap starting.
I20260925 09:16:57.558246 18154 tablet_bootstrap.cc:654] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:57.559521 18155 tablet_bootstrap.cc:654] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2: Neither blocks nor log segments found. Creating new log.
I20260925 09:16:57.559556 18154 log.cc:824] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:57.560496 18155 log.cc:824] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:57.562404 18154 tablet_bootstrap.cc:492] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0: No bootstrap required, opened a new log
I20260925 09:16:57.562677 18154 ts_tablet_manager.cc:1397] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 09:16:57.563021 18155 tablet_bootstrap.cc:492] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2: No bootstrap required, opened a new log
I20260925 09:16:57.563263 18155 ts_tablet_manager.cc:1397] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 09:16:57.570298 18154 raft_consensus.cc:348] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:16:57.570353 18155 raft_consensus.cc:348] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:16:57.570706 18154 raft_consensus.cc:374] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:57.570714 18155 raft_consensus.cc:374] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:16:57.570818 18155 raft_consensus.cc:729] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e88e973884ba4792941c82bf9a32c7d2, State: Initialized, Role: FOLLOWER
I20260925 09:16:57.570824 18154 raft_consensus.cc:729] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f71c11460c0447291b0e4d69de7afb0, State: Initialized, Role: FOLLOWER
I20260925 09:16:57.571427 18155 consensus_queue.cc:260] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:16:57.571434 18154 consensus_queue.cc:260] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:16:57.572654 18137 heartbeater.cc:503] Master 127.3.19.126:38513 was elected leader, sending a full tablet report...
I20260925 09:16:57.572654 18006 heartbeater.cc:503] Master 127.3.19.126:38513 was elected leader, sending a full tablet report...
I20260925 09:16:57.573382 18155 ts_tablet_manager.cc:1428] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260925 09:16:57.573449 18154 ts_tablet_manager.cc:1428] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260925 09:16:57.704308 18007 tablet.cc:2369] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:16:57.711771 18138 tablet.cc:2369] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:16:57.736527 17872 tablet.cc:2369] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:16:57.810487 18158 raft_consensus.cc:482] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:16:57.810835 18158 raft_consensus.cc:504] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:16:57.812630 18158 leader_election.cc:288] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087), 0b9dae99899444c0b8a3c91a54d383f9 (127.3.19.65:44861)
I20260925 09:16:57.819895 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.813079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:52077 (local address 127.3.19.67:45087)
0925 09:16:57.813631 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:57.813638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:57.814438 (+ 800us) server_negotiation.cc:407] Connection header received
0925 09:16:57.814546 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:57.814551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:57.814634 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:57.814754 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:57.815358 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.816155 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.818004 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.818773 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.818868 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.819009 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:57.819501 (+ 492us) server_negotiation.cc:299] Negotiation successful
0925 09:16:57.819622 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396}
I20260925 09:16:57.819924 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.813187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:58691 (local address 127.3.19.65:44861)
0925 09:16:57.814038 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:57.814046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:57.814071 (+ 25us) server_negotiation.cc:407] Connection header received
0925 09:16:57.814152 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:57.814157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:57.814253 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:57.814414 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:16:57.815174 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.815995 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.817991 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.818812 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.818981 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.819228 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:57.819467 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 09:16:57.819603 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":525,"threads_started":1}
I20260925 09:16:57.820032 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.812923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.3.19.66:58691)
0925 09:16:57.813706 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:57.813727 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:57.813915 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:57.814474 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:57.814481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:57.814538 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:57.815005 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:57.815016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.816144 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.816149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:57.817839 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:57.817850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.819077 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.819082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:57.819273 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.819313 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 09:16:57.819429 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 09:16:57.819573 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":93,"thread_start_us":446,"threads_started":1}
I20260925 09:16:57.820089 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.813097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:45087 (local address 127.3.19.66:52077)
0925 09:16:57.814284 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:57.814302 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:16:57.814431 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:16:57.814775 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:16:57.814778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:16:57.814797 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:16:57.815193 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:57.815206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.816298 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.816305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:57.817829 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:16:57.817842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.819015 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.819021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:16:57.819246 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.819308 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 09:16:57.819455 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 09:16:57.819586 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":42,"thread_start_us":874,"threads_started":1}
I20260925 09:16:57.821007 17825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e744337f7834c9185e7e91fb182d49a" candidate_uuid: "0f71c11460c0447291b0e4d69de7afb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9dae99899444c0b8a3c91a54d383f9" is_pre_election: true
I20260925 09:16:57.821065 18092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e744337f7834c9185e7e91fb182d49a" candidate_uuid: "0f71c11460c0447291b0e4d69de7afb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e88e973884ba4792941c82bf9a32c7d2" is_pre_election: true
I20260925 09:16:57.821614 17825 raft_consensus.cc:2454] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f71c11460c0447291b0e4d69de7afb0 in term 0.
I20260925 09:16:57.821651 18092 raft_consensus.cc:2454] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f71c11460c0447291b0e4d69de7afb0 in term 0.
I20260925 09:16:57.822386 17897 leader_election.cc:302] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b9dae99899444c0b8a3c91a54d383f9, 0f71c11460c0447291b0e4d69de7afb0, e88e973884ba4792941c82bf9a32c7d2; no voters:
I20260925 09:16:57.822886 18158 raft_consensus.cc:2781] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:16:57.823040 18158 raft_consensus.cc:482] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:16:57.823149 18158 raft_consensus.cc:3037] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:57.825389 18158 raft_consensus.cc:504] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:16:57.826081 18158 leader_election.cc:288] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [CANDIDATE]: Term 1 election: Requested vote from peers e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087), 0b9dae99899444c0b8a3c91a54d383f9 (127.3.19.65:44861)
I20260925 09:16:57.826649 18092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e744337f7834c9185e7e91fb182d49a" candidate_uuid: "0f71c11460c0447291b0e4d69de7afb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e88e973884ba4792941c82bf9a32c7d2"
I20260925 09:16:57.826680 17825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e744337f7834c9185e7e91fb182d49a" candidate_uuid: "0f71c11460c0447291b0e4d69de7afb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9dae99899444c0b8a3c91a54d383f9"
I20260925 09:16:57.826886 18092 raft_consensus.cc:3037] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:57.826910 17825 raft_consensus.cc:3037] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:16:57.830175 17825 raft_consensus.cc:2454] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f71c11460c0447291b0e4d69de7afb0 in term 1.
I20260925 09:16:57.830171 18092 raft_consensus.cc:2454] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f71c11460c0447291b0e4d69de7afb0 in term 1.
I20260925 09:16:57.830696 17896 leader_election.cc:302] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b9dae99899444c0b8a3c91a54d383f9, 0f71c11460c0447291b0e4d69de7afb0, e88e973884ba4792941c82bf9a32c7d2; no voters:
I20260925 09:16:57.831086 18158 raft_consensus.cc:2781] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:16:57.831548 18158 raft_consensus.cc:686] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [term 1 LEADER]: Becoming Leader. State: Replica: 0f71c11460c0447291b0e4d69de7afb0, State: Running, Role: LEADER
I20260925 09:16:57.832116 18158 consensus_queue.cc:237] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:16:57.837425 17680 catalog_manager.cc:5495] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66). New cstate: current_term: 1 leader_uuid: "0f71c11460c0447291b0e4d69de7afb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } health_report { overall_health: UNKNOWN } } }
I20260925 09:16:57.906876 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.901519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.3.19.66:38167)
0925 09:16:57.901710 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:57.901715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:57.901733 (+ 18us) server_negotiation.cc:407] Connection header received
0925 09:16:57.901825 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:57.901829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:57.901927 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:57.902061 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:16:57.902433 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.903167 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.904696 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.905095 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.905185 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.905734 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:16:57.906269 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:16:57.906447 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:57.906535 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 09:16:57.906610 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 09:16:57.915104 17825 raft_consensus.cc:1261] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Refusing update from remote peer 0f71c11460c0447291b0e4d69de7afb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:16:57.915109 18092 raft_consensus.cc:1261] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2 [term 1 FOLLOWER]: Refusing update from remote peer 0f71c11460c0447291b0e4d69de7afb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:16:57.916145 18163 consensus_queue.cc:1035] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:16:57.916494 18158 consensus_queue.cc:1035] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:16:57.925979 18165 mvcc.cc:205] Tried to move back new op lower bound from 7333182742170271744 to 7333182741844344832. Current Snapshot: MvccSnapshot[applied={T|T < 7333182742170271744}]
I20260925 09:16:57.930887 18167 mvcc.cc:205] Tried to move back new op lower bound from 7333182742170271744 to 7333182741844344832. Current Snapshot: MvccSnapshot[applied={T|T < 7333182742170271744}]
I20260925 09:16:57.944521 17680 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:16:57.944834 17680 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:16:57.948624 17680 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Sending DeleteTablet for 3 replicas of tablet 2e744337f7834c9185e7e91fb182d49a
I20260925 09:16:57.949998 17805 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e744337f7834c9185e7e91fb182d49a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:16:57 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:34618
I20260925 09:16:57.949954 18072 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e744337f7834c9185e7e91fb182d49a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:16:57 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:38160
I20260925 09:16:57.950047 17941 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e744337f7834c9185e7e91fb182d49a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:16:57 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:43446
I20260925 09:16:57.951020 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 17719
I20260925 09:16:57.959609 18176 tablet_replica.cc:330] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2: stopping tablet replica
I20260925 09:16:57.960834 18176 raft_consensus.cc:2229] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:16:57.962136 18176 raft_consensus.cc:2258] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:16:57.963145 18177 tablet_replica.cc:330] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0: stopping tablet replica
I20260925 09:16:57.964129 18177 raft_consensus.cc:2229] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:16:57.964936 18177 raft_consensus.cc:2258] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:16:57.967422 18177 ts_tablet_manager.cc:1905] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:16:57.982347 18176 ts_tablet_manager.cc:1905] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:16:57.986896 18177 ts_tablet_manager.cc:1918] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:16:57.989683 18177 log.cc:1196] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/2e744337f7834c9185e7e91fb182d49a
I20260925 09:16:57.990412 18177 ts_tablet_manager.cc:1939] T 2e744337f7834c9185e7e91fb182d49a P 0f71c11460c0447291b0e4d69de7afb0: Deleting consensus metadata
I20260925 09:16:57.992708 17668 catalog_manager.cc:4841] TS 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167): tablet 2e744337f7834c9185e7e91fb182d49a (table test-table [id=e551f2c4b80f4805bc5ef1cfc23fb0f3]) successfully deleted
I20260925 09:16:58.002496 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:57.956287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.3.19.65:44861)
0925 09:16:57.957142 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:57.957151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:57.957176 (+ 25us) server_negotiation.cc:407] Connection header received
0925 09:16:57.957271 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:57.957280 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:57.957386 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:57.969436 (+ 12050us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:57.969730 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.971234 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.976539 (+ 5305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:57.977580 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:57.984187 (+ 6607us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:57.990848 (+ 6661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:57.990859 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:57.990868 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:57.990930 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:57.996809 (+ 5879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:57.998142 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:57.998151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:57.998157 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:57.998398 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:57.998998 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:57.999005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:57.999011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:57.999499 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:58.001415 (+ 1916us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:58.001597 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 09:16:58.001970 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642}
I20260925 09:16:58.003818 17865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:58.007803 18176 ts_tablet_manager.cc:1918] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:16:58.008023 18176 log.cc:1196] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/2e744337f7834c9185e7e91fb182d49a
I20260925 09:16:58.008659 18176 ts_tablet_manager.cc:1939] T 2e744337f7834c9185e7e91fb182d49a P e88e973884ba4792941c82bf9a32c7d2: Deleting consensus metadata
I20260925 09:16:58.010902 17669 catalog_manager.cc:4841] TS e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087): tablet 2e744337f7834c9185e7e91fb182d49a (table test-table [id=e551f2c4b80f4805bc5ef1cfc23fb0f3]) successfully deleted
I20260925 09:16:58.127285 18175 tablet_replica.cc:330] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9: stopping tablet replica
I20260925 09:16:58.127594 18175 raft_consensus.cc:2229] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:16:58.127846 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 17719
I20260925 09:16:58.127871 18175 raft_consensus.cc:2258] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:16:58.153131 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 17875
W20260925 09:16:58.153302 17668 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.19.65:44861 (error 108)
I20260925 09:16:58.157027 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:58.156272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34640)
0925 09:16:58.156741 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:58.156848 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":262,"threads_started":1}
W20260925 09:16:58.158491 17668 catalog_manager.cc:4584] TS 0b9dae99899444c0b8a3c91a54d383f9 (127.3.19.65:44861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e744337f7834c9185e7e91fb182d49a: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
I20260925 09:16:58.170701 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:58.154024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43464 (local address 127.3.19.66:38167)
0925 09:16:58.154914 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:58.154924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:58.154953 (+ 29us) server_negotiation.cc:407] Connection header received
0925 09:16:58.155037 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:58.155045 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:58.155165 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:58.155303 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:58.157194 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:58.158182 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:58.160129 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:58.160586 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:58.160685 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:58.164072 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:58.164087 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:58.164103 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:58.164145 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:58.167427 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:58.168423 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:58.168430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:58.168433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:58.168601 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:58.169050 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:58.169053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:58.169055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:58.169527 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:58.169809 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:58.170145 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 09:16:58.170322 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697}
I20260925 09:16:58.171486 18001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:58.212220 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:58.211609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34642)
0925 09:16:58.211914 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:58.212043 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260925 09:16:58.213522 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:58.213070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34654)
0925 09:16:58.213265 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:58.213354 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 09:16:58.274567 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:58.273978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34664)
0925 09:16:58.274251 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:58.274363 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 09:16:58.275874 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:58.275422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34666)
0925 09:16:58.275591 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:58.275688 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 09:16:58.289410 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 17875
I20260925 09:16:58.309965 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 18009
I20260925 09:16:58.323805 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:58.310928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.3.19.67:45087)
0925 09:16:58.311140 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 09:16:58.311146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:16:58.311166 (+ 20us) server_negotiation.cc:407] Connection header received
0925 09:16:58.311382 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:16:58.311386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:16:58.311451 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:16:58.311559 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:16:58.312953 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:58.313734 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:58.315033 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:16:58.315414 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:16:58.315548 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:16:58.318232 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:16:58.318246 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:16:58.318262 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:16:58.318300 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:16:58.320848 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:58.321663 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:58.321675 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:58.321683 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:58.321822 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:16:58.322216 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:16:58.322222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:16:58.322229 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:16:58.322590 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:16:58.322901 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 09:16:58.323229 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 09:16:58.323453 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 09:16:58.324496 18132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:58.357262 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:58.356717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34682)
0925 09:16:58.356964 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:58.357065 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 09:16:58.358497 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:58.358059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34694)
0925 09:16:58.358286 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:58.358358 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 09:16:58.416177 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 18009
I20260925 09:16:58.433874 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 17649
I20260925 09:16:58.435086 17712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:16:58.549809 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:58.548724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:44861 (local address 127.0.0.1:34706)
0925 09:16:58.549327 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:58.549465 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 09:16:58.549932 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 17649
I20260925 09:16:58.573088 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:38513
--webserver_interface=127.3.19.126
--webserver_port=33129
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:16:58.919883 18184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:58.920192 18184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:58.920291 18184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:58.927639 18184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:16:58.927768 18184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:58.927824 18184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:58.927891 18184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:16:58.927937 18184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:16:58.938087 18184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:38513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=33129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:58.939459 18184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:58.940711 18184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:58.953560 18192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:58.953599 18190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:58.953647 18189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:58.959936 18184 server_base.cc:1034] running on GCE node
I20260925 09:16:58.960876 18184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:58.962513 18184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:58.963761 18184 hybrid_clock.cc:648] HybridClock initialized: now 1790327818963706 us; error 53 us; skew 500 ppm
I20260925 09:16:58.964133 18184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:58 dist-test-slave-l6qh krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327818, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:58.967420 18184 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:16:58.969210 18184 webserver.cc:460] Webserver started at http://127.3.19.126:33129/ using document root <none> and password file <none>
I20260925 09:16:58.970012 18184 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:58.970139 18184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:58.975729 18184 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 09:16:58.978709 18199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:58.980165 18184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:16:58.980369 18184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "470394a6942e419d98230eb3deeb5703"
format_stamp: "Formatted at 2026-09-25 09:16:52 on dist-test-slave-l6qh"
server_key: "f828e49ea8a502628cbb21532d0730c5"
server_key_iv: "eabbb67e767b544c91345216b91c04e0"
server_key_version: "encryptionkey@0"
I20260925 09:16:58.981545 18184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:59.020376 18184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:59.024875 18184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:59.025182 18184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:59.044965 18184 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:38513
I20260925 09:16:59.045002 18250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:38513 every 8 connection(s)
I20260925 09:16:59.046583 18184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:16:59.049405 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 18184
I20260925 09:16:59.050232 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:44861
--local_ip_for_outbound_sockets=127.3.19.65
--tserver_master_addrs=127.3.19.126:38513
--webserver_port=33303
--webserver_interface=127.3.19.65
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:16:59.054000 18251 sys_catalog.cc:263] Verifying existing consensus state
I20260925 09:16:59.057462 18251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Bootstrap starting.
I20260925 09:16:59.089483 18251 log.cc:824] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:59.111392 18251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 09:16:59.112128 18251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703: Bootstrap complete.
I20260925 09:16:59.127336 18251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } }
I20260925 09:16:59.128647 18251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 470394a6942e419d98230eb3deeb5703, State: Initialized, Role: FOLLOWER
I20260925 09:16:59.129484 18251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } }
I20260925 09:16:59.129771 18251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:16:59.129899 18251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:16:59.130117 18251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 2 FOLLOWER]: Advancing to term 3
I20260925 09:16:59.135851 18251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } }
I20260925 09:16:59.136502 18251 leader_election.cc:302] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 470394a6942e419d98230eb3deeb5703; no voters:
I20260925 09:16:59.137219 18251 leader_election.cc:288] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 09:16:59.137475 18255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 09:16:59.138586 18255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [term 3 LEADER]: Becoming Leader. State: Replica: 470394a6942e419d98230eb3deeb5703, State: Running, Role: LEADER
I20260925 09:16:59.139243 18255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } }
I20260925 09:16:59.139457 18251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:16:59.141902 18257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 470394a6942e419d98230eb3deeb5703. Latest consensus state: current_term: 3 leader_uuid: "470394a6942e419d98230eb3deeb5703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } } }
I20260925 09:16:59.142349 18257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:59.141897 18256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "470394a6942e419d98230eb3deeb5703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470394a6942e419d98230eb3deeb5703" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 38513 } } }
I20260925 09:16:59.142505 18256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470394a6942e419d98230eb3deeb5703 [sys.catalog]: This master's current role is: LEADER
I20260925 09:16:59.149748 18262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:16:59.159268 18262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:16:59.160889 18262 catalog_manager.cc:1259] Loaded cluster ID: 4a543d38b1074741a2a3db8701600a92
I20260925 09:16:59.161079 18262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 09:16:59.163249 18262 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fef295479ad google::LogMessage::Fail() at ??:0
@ 0x7fef2954a88a google::LogMessage::SendToLog() at ??:0
@ 0x7fef295474de google::LogMessage::Flush() at ??:0
@ 0x7fef29548a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260925 09:16:59.172153 18272 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fef3ef7e512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fef3ef7b976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fef3f003610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fef3f003439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fef2c193345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fef2c19691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fef2c196799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fef2c171eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fef2f44b6db start_thread at ??:0
@ 0x7fef283e161f clone at ??:0
W20260925 09:16:59.482126 18253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:59.482421 18253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:59.482597 18253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:59.490314 18253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:59.490434 18253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:59.490613 18253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:16:59.500819 18253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:44861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=33303
--tserver_master_addrs=127.3.19.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:16:59.502238 18253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:16:59.503640 18253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:16:59.517802 18292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:59.517800 18295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:16:59.517784 18293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:16:59.518671 18253 server_base.cc:1034] running on GCE node
I20260925 09:16:59.519637 18253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:16:59.520705 18253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:16:59.521944 18253 hybrid_clock.cc:648] HybridClock initialized: now 1790327819521869 us; error 73 us; skew 500 ppm
I20260925 09:16:59.522264 18253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:16:59 dist-test-slave-l6qh krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:16:59.525745 18253 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:16:59.527333 18253 webserver.cc:460] Webserver started at http://127.3.19.65:33303/ using document root <none> and password file <none>
I20260925 09:16:59.528105 18253 fs_manager.cc:359] Metadata directory not provided
I20260925 09:16:59.528219 18253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:16:59.534125 18253 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 09:16:59.537209 18302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:16:59.538836 18253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 09:16:59.539058 18253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "0b9dae99899444c0b8a3c91a54d383f9"
format_stamp: "Formatted at 2026-09-25 09:16:53 on dist-test-slave-l6qh"
server_key: "b12358e4d5775253bf99edce05c8aeda"
server_key_iv: "41bb66723f1ad47c649f6a29894da445"
server_key_version: "encryptionkey@0"
I20260925 09:16:59.540141 18253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:16:59.565478 18253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:16:59.569741 18253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:16:59.569986 18253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:16:59.570986 18253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:16:59.572854 18309 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 09:16:59.578897 18253 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 09:16:59.579066 18253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260925 09:16:59.579222 18253 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 09:16:59.582581 18253 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 09:16:59.582705 18253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.002s
I20260925 09:16:59.583043 18309 tablet_bootstrap.cc:492] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9: Bootstrap starting.
I20260925 09:16:59.611235 18309 log.cc:824] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9: Log is configured to *not* fsync() on all Append() calls
I20260925 09:16:59.625028 18253 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:44861
I20260925 09:16:59.625065 18416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:44861 every 8 connection(s)
I20260925 09:16:59.625494 18309 tablet_bootstrap.cc:492] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 09:16:59.626060 18309 tablet_bootstrap.cc:492] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9: Bootstrap complete.
I20260925 09:16:59.626601 18253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:16:59.627321 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 18253
I20260925 09:16:59.627622 18309 ts_tablet_manager.cc:1397] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9: Time spent bootstrapping tablet: real 0.045s user 0.026s sys 0.016s
I20260925 09:16:59.628324 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:38167
--local_ip_for_outbound_sockets=127.3.19.66
--tserver_master_addrs=127.3.19.126:38513
--webserver_port=40087
--webserver_interface=127.3.19.66
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:16:59.632376 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:59.629745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:59645)
0925 09:16:59.630715 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:59.631018 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":426,"threads_started":1}
W20260925 09:16:59.633102 18417 heartbeater.cc:644] Failed to heartbeat to 127.3.19.126:38513 (0 consecutive failures): Network error: Failed to ping master at 127.3.19.126:38513: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
I20260925 09:16:59.634080 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:59.633669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:51783)
0925 09:16:59.633835 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:59.633921 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 09:16:59.639190 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:59.634779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:58889)
0925 09:16:59.638570 (+ 3791us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:59.638837 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3592,"thread_start_us":3277,"threads_started":1}
W20260925 09:16:59.639564 18417 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 09:16:59.642297 18309 raft_consensus.cc:348] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:16:59.643296 18309 raft_consensus.cc:729] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9dae99899444c0b8a3c91a54d383f9, State: Initialized, Role: FOLLOWER
I20260925 09:16:59.644071 18309 consensus_queue.cc:260] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:16:59.646162 18309 ts_tablet_manager.cc:1428] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9: Time spent starting tablet: real 0.018s user 0.014s sys 0.002s
I20260925 09:16:59.646183 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:59.645766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:57665)
0925 09:16:59.645947 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:59.646038 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 09:16:59.647317 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:16:59.646913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:48933)
0925 09:16:59.647089 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 09:16:59.647181 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 09:16:59.978794 18421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:16:59.979151 18421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:16:59.979341 18421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:16:59.987473 18421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:16:59.987615 18421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:16:59.987794 18421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:16:59.998783 18421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:38167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=40087
--tserver_master_addrs=127.3.19.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:17:00.000298 18421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:17:00.001983 18421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:17:00.019472 18427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:00.019472 18428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:00.019472 18430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:17:00.020754 18421 server_base.cc:1034] running on GCE node
I20260925 09:17:00.021617 18421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:17:00.022785 18421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:17:00.024052 18421 hybrid_clock.cc:648] HybridClock initialized: now 1790327820024004 us; error 53 us; skew 500 ppm
I20260925 09:17:00.024454 18421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:17:00 dist-test-slave-l6qh krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:17:00.028671 18421 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:17:00.030794 18421 webserver.cc:460] Webserver started at http://127.3.19.66:40087/ using document root <none> and password file <none>
I20260925 09:17:00.031656 18421 fs_manager.cc:359] Metadata directory not provided
I20260925 09:17:00.031821 18421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:17:00.037797 18421 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 09:17:00.041400 18437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:00.043203 18421 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 09:17:00.043469 18421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "0f71c11460c0447291b0e4d69de7afb0"
format_stamp: "Formatted at 2026-09-25 09:16:54 on dist-test-slave-l6qh"
server_key: "c55832ba1f3c50fe3ef8096dd7b5e380"
server_key_iv: "9439183cb8e46dfd6661a850339edfa6"
server_key_version: "encryptionkey@0"
I20260925 09:17:00.044766 18421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:17:00.073180 18421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:17:00.078115 18421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:17:00.078378 18421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:17:00.079427 18421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:17:00.080839 18421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:17:00.080928 18421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:00.081004 18421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:17:00.081063 18421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:00.118589 18421 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:38167
I20260925 09:17:00.118690 18549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:38167 every 8 connection(s)
I20260925 09:17:00.120502 18421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:17:00.122921 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 18421
I20260925 09:17:00.123593 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:45087
--local_ip_for_outbound_sockets=127.3.19.67
--tserver_master_addrs=127.3.19.126:38513
--webserver_port=45257
--webserver_interface=127.3.19.67
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:17:00.126816 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:00.124095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:55299)
0925 09:17:00.125066 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:00.125436 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":355,"threads_started":1}
W20260925 09:17:00.127727 18550 heartbeater.cc:644] Failed to heartbeat to 127.3.19.126:38513 (0 consecutive failures): Network error: Failed to ping master at 127.3.19.126:38513: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
I20260925 09:17:00.128943 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:00.128427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:54833)
0925 09:17:00.128652 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:00.128762 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 09:17:00.130144 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:00.129649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:35675)
0925 09:17:00.129847 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:00.129972 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 09:17:00.130422 18550 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 09:17:00.472901 18553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:17:00.473193 18553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:17:00.473388 18553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:17:00.480700 18553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:17:00.480822 18553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:17:00.480938 18553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:17:00.491252 18553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:45087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=45257
--tserver_master_addrs=127.3.19.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:17:00.492601 18553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:17:00.493934 18553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:17:00.507546 18561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:00.507655 18559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:00.507570 18558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:17:00.508327 18553 server_base.cc:1034] running on GCE node
I20260925 09:17:00.509008 18553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:17:00.510036 18553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:17:00.511248 18553 hybrid_clock.cc:648] HybridClock initialized: now 1790327820511205 us; error 28 us; skew 500 ppm
I20260925 09:17:00.511598 18553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:17:00 dist-test-slave-l6qh krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:17:00.514981 18553 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:17:00.516474 18553 webserver.cc:460] Webserver started at http://127.3.19.67:45257/ using document root <none> and password file <none>
I20260925 09:17:00.517202 18553 fs_manager.cc:359] Metadata directory not provided
I20260925 09:17:00.517411 18553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:17:00.523037 18553 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 09:17:00.526068 18568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:00.527590 18553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:17:00.527804 18553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "e88e973884ba4792941c82bf9a32c7d2"
format_stamp: "Formatted at 2026-09-25 09:16:54 on dist-test-slave-l6qh"
server_key: "66da70c1999f9abb57b38cad06798c4b"
server_key_iv: "0fdba63c966fdc483d14923d99f2bd9b"
server_key_version: "encryptionkey@0"
I20260925 09:17:00.528883 18553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:17:00.551707 18553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:17:00.556463 18553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:17:00.556766 18553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:17:00.557937 18553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:17:00.559404 18553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:17:00.559489 18553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:00.559654 18553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:17:00.559754 18553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:00.592914 18553 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:45087
I20260925 09:17:00.592996 18680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:45087 every 8 connection(s)
I20260925 09:17:00.594403 18553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:17:00.599532 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:00.597618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:49973)
0925 09:17:00.598400 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:00.598633 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":291,"threads_started":1}
W20260925 09:17:00.600245 18681 heartbeater.cc:644] Failed to heartbeat to 127.3.19.126:38513 (0 consecutive failures): Network error: Failed to ping master at 127.3.19.126:38513: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
I20260925 09:17:00.601061 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:00.600715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:49629)
0925 09:17:00.600882 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:00.600952 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 09:17:00.601716 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 18553
I20260925 09:17:00.602102 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:00.601646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:38443)
0925 09:17:00.601859 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:00.601955 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260925 09:17:00.602453 18681 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 09:17:00.649324 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:00.648244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:46057)
0925 09:17:00.648833 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:00.648939 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":236,"threads_started":1}
I20260925 09:17:01.131865 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:01.131018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:55809)
0925 09:17:01.131511 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:01.131606 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":237,"threads_started":1}
I20260925 09:17:01.365427 18687 raft_consensus.cc:482] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:17:01.365728 18687 raft_consensus.cc:504] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:17:01.367403 18687 leader_election.cc:288] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167), e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087)
Sep 25 09:17:01 dist-test-slave-l6qh krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
Sep 25 09:17:01 dist-test-slave-l6qh krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:17:01.402853 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:01.368028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:54277 (local address 127.3.19.66:38167)
0925 09:17:01.381249 (+ 13221us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:01.381265 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:01.381320 (+ 55us) server_negotiation.cc:407] Connection header received
0925 09:17:01.382964 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:01.382972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:01.383324 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:01.383694 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:01.385864 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:01.387255 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:01.389343 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:01.389984 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:01.390197 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:01.394788 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:01.394803 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:01.394827 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:01.394887 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:01.398759 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:01.399735 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:01.399743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:01.399751 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:01.399916 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:01.400408 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:01.400420 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:01.400423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:01.400969 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:01.401186 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:01.402047 (+ 861us) server_negotiation.cc:299] Negotiation successful
0925 09:17:01.402429 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13004,"thread_start_us":12812,"threads_started":1}
I20260925 09:17:01.402860 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:01.368205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:34361 (local address 127.3.19.67:45087)
0925 09:17:01.379569 (+ 11364us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:01.379596 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:01.381132 (+ 1536us) server_negotiation.cc:407] Connection header received
0925 09:17:01.382900 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:01.382910 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:01.383325 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:01.383690 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:01.385870 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:01.387247 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:01.389270 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:01.389945 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:01.390172 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:01.394472 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:01.394480 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:01.394497 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:01.394569 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:01.398743 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:01.399762 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:01.399769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:01.399775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:01.399938 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:01.400432 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:01.400436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:01.400438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:01.400968 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:01.401187 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:01.402021 (+ 834us) server_negotiation.cc:299] Negotiation successful
0925 09:17:01.402408 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11044,"thread_start_us":11417,"threads_started":1}
I20260925 09:17:01.402878 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:01.367615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:38167 (local address 127.3.19.65:54277)
0925 09:17:01.380347 (+ 12732us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:01.380407 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:01.381163 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:01.383858 (+ 2695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:01.383869 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:01.384780 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:17:01.385641 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:01.385665 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:01.387488 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:01.387497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:01.389072 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:01.389086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:01.390225 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:01.390233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:01.390542 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:01.391423 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:17:01.391450 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:17:01.394473 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:17:01.398946 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:01.398950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:01.398959 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:01.399561 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:01.400041 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:01.400046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:01.400052 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:01.400269 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:01.401123 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:17:01.401131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:17:01.401509 (+ 378us) client_negotiation.cc:769] Sending connection context
0925 09:17:01.401794 (+ 285us) client_negotiation.cc:240] Negotiation successful
0925 09:17:01.402415 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12592,"mutex_wait_us":261,"spinlock_wait_cycles":105984,"thread_start_us":12807,"threads_started":1}
I20260925 09:17:01.403049 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:01.367751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:45087 (local address 127.3.19.65:34361)
0925 09:17:01.380646 (+ 12895us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:01.380662 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:01.381130 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:01.383858 (+ 2728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:01.383869 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:01.384780 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:17:01.385641 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:01.385665 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:01.387464 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:01.387468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:01.389074 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:01.389086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:01.390170 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:01.390181 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:01.390471 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:01.391493 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:17:01.391511 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:17:01.394127 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:17:01.398937 (+ 4810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:01.398944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:01.398958 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:01.399573 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:01.400080 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:01.400083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:01.400085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:01.400278 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:01.401128 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:17:01.401132 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:17:01.401509 (+ 377us) client_negotiation.cc:769] Sending connection context
0925 09:17:01.401794 (+ 285us) client_negotiation.cc:240] Negotiation successful
0925 09:17:01.402585 (+ 791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12794,"mutex_wait_us":470,"spinlock_wait_cycles":28032,"thread_start_us":12294,"threads_started":1}
W20260925 09:17:01.406548 18305 leader_election.cc:341] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167): Not found: Tablet not found: 2e744337f7834c9185e7e91fb182d49a
W20260925 09:17:01.406862 18306 leader_election.cc:341] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087): Not found: Tablet not found: 2e744337f7834c9185e7e91fb182d49a
I20260925 09:17:01.407027 18306 leader_election.cc:302] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9dae99899444c0b8a3c91a54d383f9; no voters: 0f71c11460c0447291b0e4d69de7afb0, e88e973884ba4792941c82bf9a32c7d2
I20260925 09:17:01.407470 18687 raft_consensus.cc:2726] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 09:17:01.603911 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:01.602998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:42659)
0925 09:17:01.603503 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:01.603655 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":256,"threads_started":1}
I20260925 09:17:01.651039 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:01.650410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:45037)
0925 09:17:01.650639 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:01.650808 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 09:17:02.133850 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:02.132989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:33585)
0925 09:17:02.133519 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:02.133662 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":312,"threads_started":1}
I20260925 09:17:02.605739 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:02.604902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:49705)
0925 09:17:02.605457 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:02.605554 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":268,"threads_started":1}
I20260925 09:17:02.652868 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:02.652001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:57793)
0925 09:17:02.652563 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:02.652668 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":255,"threads_started":1}
I20260925 09:17:03.135516 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:03.134725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:33697)
0925 09:17:03.135242 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:03.135349 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":267,"threads_started":1}
I20260925 09:17:03.206977 18697 raft_consensus.cc:482] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:17:03.207146 18697 raft_consensus.cc:504] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:17:03.207736 18697 leader_election.cc:288] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167), e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087)
W20260925 09:17:03.208814 18306 leader_election.cc:341] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087): Not found: Tablet not found: 2e744337f7834c9185e7e91fb182d49a
W20260925 09:17:03.208988 18305 leader_election.cc:341] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167): Not found: Tablet not found: 2e744337f7834c9185e7e91fb182d49a
I20260925 09:17:03.209142 18305 leader_election.cc:302] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9dae99899444c0b8a3c91a54d383f9; no voters: 0f71c11460c0447291b0e4d69de7afb0, e88e973884ba4792941c82bf9a32c7d2
I20260925 09:17:03.209432 18697 raft_consensus.cc:2726] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 09:17:03.607287 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:03.606576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:32947)
0925 09:17:03.607040 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:03.607120 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":253,"threads_started":1}
I20260925 09:17:03.654637 18699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:03.653870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:46403)
0925 09:17:03.654328 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:03.654444 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":227,"threads_started":1}
I20260925 09:17:04.136989 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:04.136313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:57551)
0925 09:17:04.136744 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:04.136823 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":213,"threads_started":1}
I20260925 09:17:04.608947 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:04.608202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:50073)
0925 09:17:04.608688 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:04.608777 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":267,"threads_started":1}
I20260925 09:17:04.656224 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:04.655598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:52875)
0925 09:17:04.655980 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:04.656052 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":202,"threads_started":1}
I20260925 09:17:04.777762 18703 raft_consensus.cc:482] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:17:04.777978 18703 raft_consensus.cc:504] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:17:04.778671 18703 leader_election.cc:288] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167), e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087)
W20260925 09:17:04.779560 18305 leader_election.cc:341] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167): Not found: Tablet not found: 2e744337f7834c9185e7e91fb182d49a
W20260925 09:17:04.779728 18306 leader_election.cc:341] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087): Not found: Tablet not found: 2e744337f7834c9185e7e91fb182d49a
I20260925 09:17:04.779896 18306 leader_election.cc:302] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9dae99899444c0b8a3c91a54d383f9; no voters: 0f71c11460c0447291b0e4d69de7afb0, e88e973884ba4792941c82bf9a32c7d2
I20260925 09:17:04.780167 18703 raft_consensus.cc:2726] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 09:17:05.138509 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:05.137830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:55875)
0925 09:17:05.138280 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:05.138358 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":221,"threads_started":1}
I20260925 09:17:05.610532 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:05.609768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:33431)
0925 09:17:05.610272 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:05.610359 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":261,"threads_started":1}
I20260925 09:17:05.658028 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:05.657116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:38437)
0925 09:17:05.657732 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:05.657855 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":399,"threads_started":1}
I20260925 09:17:06.140106 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:06.139397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:35079)
0925 09:17:06.139854 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:06.139929 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":263,"threads_started":1}
I20260925 09:17:06.595413 18708 raft_consensus.cc:482] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:17:06.595619 18708 raft_consensus.cc:504] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:17:06.596232 18708 leader_election.cc:288] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167), e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087)
W20260925 09:17:06.597275 18306 leader_election.cc:341] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087): Not found: Tablet not found: 2e744337f7834c9185e7e91fb182d49a
W20260925 09:17:06.597508 18305 leader_election.cc:341] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167): Not found: Tablet not found: 2e744337f7834c9185e7e91fb182d49a
I20260925 09:17:06.597625 18305 leader_election.cc:302] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9dae99899444c0b8a3c91a54d383f9; no voters: 0f71c11460c0447291b0e4d69de7afb0, e88e973884ba4792941c82bf9a32c7d2
I20260925 09:17:06.597913 18708 raft_consensus.cc:2726] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 09:17:06.612126 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:06.611357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:36547)
0925 09:17:06.611866 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:06.611962 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":295,"threads_started":1}
I20260925 09:17:06.659729 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:06.658913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:42571)
0925 09:17:06.659422 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:06.659514 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":264,"threads_started":1}
I20260925 09:17:07.141781 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:07.140915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:34717)
0925 09:17:07.141488 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:07.141582 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":281,"threads_started":1}
I20260925 09:17:07.613901 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:07.613042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:55895)
0925 09:17:07.613632 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:07.613716 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":309,"threads_started":1}
I20260925 09:17:07.661294 18713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:07.660572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:36677)
0925 09:17:07.661002 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:07.661080 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":237,"threads_started":1}
I20260925 09:17:08.143270 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:08.142624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:33025)
0925 09:17:08.143025 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:08.143105 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":218,"threads_started":1}
I20260925 09:17:08.615444 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:08.614763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:51297)
0925 09:17:08.615176 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:08.615266 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":217,"threads_started":1}
I20260925 09:17:08.662906 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:08.662212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:48127)
0925 09:17:08.662637 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:08.662726 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":214,"threads_started":1}
I20260925 09:17:09.145090 18717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:09.144192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:51025)
0925 09:17:09.144718 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:09.144863 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":216,"threads_started":1}
I20260925 09:17:09.617091 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:09.616380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:54819)
0925 09:17:09.616852 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:09.616940 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":233,"threads_started":1}
I20260925 09:17:09.664597 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:09.663923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:49603)
0925 09:17:09.664338 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:09.664436 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":214,"threads_started":1}
I20260925 09:17:10.146828 18720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:10.146037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:40839)
0925 09:17:10.146474 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:10.146552 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":232,"threads_started":1}
I20260925 09:17:10.618616 18721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:10.617944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:40675)
0925 09:17:10.618366 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:10.618441 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":253,"threads_started":1}
I20260925 09:17:10.666163 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:10.665466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:38489)
0925 09:17:10.665912 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:10.665988 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":223,"threads_started":1}
I20260925 09:17:11.148613 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:11.147774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:40457)
0925 09:17:11.148288 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:11.148378 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":222,"threads_started":1}
I20260925 09:17:11.620255 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:11.619473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:37669)
0925 09:17:11.620010 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:11.620091 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":243,"threads_started":1}
I20260925 09:17:11.667654 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:11.666996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:52897)
0925 09:17:11.667403 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:11.667477 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":201,"threads_started":1}
I20260925 09:17:11.704563 18726 raft_consensus.cc:482] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:17:11.704818 18726 raft_consensus.cc:504] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f71c11460c0447291b0e4d69de7afb0" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 38167 } } peers { permanent_uuid: "e88e973884ba4792941c82bf9a32c7d2" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 45087 } } peers { permanent_uuid: "0b9dae99899444c0b8a3c91a54d383f9" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 44861 } }
I20260925 09:17:11.705530 18726 leader_election.cc:288] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167), e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087)
W20260925 09:17:11.706499 18306 leader_election.cc:341] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e88e973884ba4792941c82bf9a32c7d2 (127.3.19.67:45087): Not found: Tablet not found: 2e744337f7834c9185e7e91fb182d49a
W20260925 09:17:11.706732 18305 leader_election.cc:341] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f71c11460c0447291b0e4d69de7afb0 (127.3.19.66:38167): Not found: Tablet not found: 2e744337f7834c9185e7e91fb182d49a
I20260925 09:17:11.706871 18305 leader_election.cc:302] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b9dae99899444c0b8a3c91a54d383f9; no voters: 0f71c11460c0447291b0e4d69de7afb0, e88e973884ba4792941c82bf9a32c7d2
I20260925 09:17:11.707134 18726 raft_consensus.cc:2726] T 2e744337f7834c9185e7e91fb182d49a P 0b9dae99899444c0b8a3c91a54d383f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 09:17:12.150197 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:12.149460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:51651)
0925 09:17:12.149903 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:12.150028 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":240,"threads_started":1}
I20260925 09:17:12.621881 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:12.621157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:51589)
0925 09:17:12.621633 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:12.621716 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":270,"threads_started":1}
I20260925 09:17:12.669337 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:12.668517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:50967)
0925 09:17:12.668975 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:12.669060 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":256,"threads_started":1}
I20260925 09:17:13.151659 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:13.150990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:40011)
0925 09:17:13.151415 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:13.151498 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":210,"threads_started":1}
I20260925 09:17:13.623469 18731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:13.622758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:55025)
0925 09:17:13.623174 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:13.623257 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":219,"threads_started":1}
I20260925 09:17:13.670871 18732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:13.670175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:43005)
0925 09:17:13.670580 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:13.670679 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":206,"threads_started":1}
I20260925 09:17:14.153316 18733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:14.152540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:49291)
0925 09:17:14.152980 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:14.153057 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":235,"threads_started":1}
I20260925 09:17:14.625180 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:14.624335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:56277)
0925 09:17:14.624919 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:14.624994 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":352,"threads_started":1}
I20260925 09:17:14.672734 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:14.672005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.65:39579)
0925 09:17:14.672443 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:14.672543 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":208,"threads_started":1}
I20260925 09:17:15.155019 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:15.154340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.66:39083)
0925 09:17:15.154769 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:15.154862 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":213,"threads_started":1}
I20260925 09:17:15.609326 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 18253
I20260925 09:17:15.623862 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:15.610652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.3.19.65:44861)
0925 09:17:15.611270 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:15.611280 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:15.611319 (+ 39us) server_negotiation.cc:407] Connection header received
0925 09:17:15.611414 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:15.611424 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:15.611748 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:15.611909 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:15.613319 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:15.614254 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:15.615409 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:15.615948 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:15.616040 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:15.618676 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:15.618687 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:15.618700 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:15.618732 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:15.621603 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:15.622249 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:15.622255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:15.622257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:15.622382 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:15.622693 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:15.622696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:15.622698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:15.623005 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:15.623250 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:15.623440 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 09:17:15.623620 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":250,"threads_started":1}
I20260925 09:17:15.625097 18412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:17:15.627281 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:15.626329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:38513 (local address 127.3.19.67:34071)
0925 09:17:15.626885 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:15.627043 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":254,"threads_started":1}
I20260925 09:17:15.721186 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 18253
I20260925 09:17:15.739358 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 18421
I20260925 09:17:15.752187 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:15.740115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.3.19.66:38167)
0925 09:17:15.740631 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:15.740636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:15.740651 (+ 15us) server_negotiation.cc:407] Connection header received
0925 09:17:15.740709 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:15.740712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:15.740773 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:15.740861 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:15.742216 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:15.743015 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:15.744089 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:15.744509 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:15.744605 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:15.747119 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:15.747123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:15.747126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:15.747154 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:15.750034 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:15.750638 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:15.750642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:15.750644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:15.750760 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:15.751081 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:15.751084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:15.751085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:15.751333 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:15.751515 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:15.751720 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 09:17:15.751887 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":300,"threads_started":1}
I20260925 09:17:15.752879 18545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:17:15.834822 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 18421
I20260925 09:17:15.852728 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 18553
I20260925 09:17:15.865484 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:15.853512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.3.19.67:45087)
0925 09:17:15.854051 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:15.854056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:15.854086 (+ 30us) server_negotiation.cc:407] Connection header received
0925 09:17:15.854140 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:15.854143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:15.854227 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:15.854314 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:15.855592 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:15.856343 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:15.857690 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:15.858097 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:15.858196 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:15.860607 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:15.860611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:15.860614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:15.860644 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:15.863078 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:15.863621 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:15.863643 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:15.863645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:15.863823 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:15.864149 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:15.864152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:15.864154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:15.864428 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:15.864637 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:15.864861 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 09:17:15.865010 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":275,"threads_started":1}
I20260925 09:17:15.866159 18676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:17:15.948550 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 18553
2026-09-25T09:17:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24802 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:17:16 dist-test-slave-l6qh krb5kdc[18747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:17:16 dist-test-slave-l6qh krb5kdc[18747](info): set up 2 sockets
Sep 25 09:17:16 dist-test-slave-l6qh krb5kdc[18747](info): commencing operation
krb5kdc: starting...
W20260925 09:17:17.034792 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:17:17 dist-test-slave-l6qh krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:17:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:17:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:17:17.266819 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46573
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:46573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:17:17.586673 18760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:17:17.586956 18760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:17:17.587049 18760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:17:17.593945 18760 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 09:17:17.594031 18760 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 09:17:17.594069 18760 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 09:17:17.594122 18760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:17:17.594183 18760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:17:17.594256 18760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:17:17.594302 18760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:17:17.594352 18760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:17:17.603753 18760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:46573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:17:17.604919 18760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:17:17.606164 18760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:17:17.618176 18765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:17.618211 18766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:17.618294 18768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:17:17.618729 18760 server_base.cc:1034] running on GCE node
I20260925 09:17:17.619491 18760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:17:17.620765 18760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:17:17.622023 18760 hybrid_clock.cc:648] HybridClock initialized: now 1790327837621972 us; error 46 us; skew 500 ppm
I20260925 09:17:17.622362 18760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:17:17 dist-test-slave-l6qh krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327837, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:17:17.629123 18760 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:17:17.630623 18760 webserver.cc:460] Webserver started at http://127.3.19.126:37393/ using document root <none> and password file <none>
I20260925 09:17:17.631346 18760 fs_manager.cc:359] Metadata directory not provided
I20260925 09:17:17.631489 18760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:17:17.631829 18760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:17:17.635174 18760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "1c76d36dcf4749e5888ae8ab5e896e43"
format_stamp: "Formatted at 2026-09-25 09:17:17 on dist-test-slave-l6qh"
server_key: "270bc535692164dbb5798141f74570bc"
server_key_iv: "dd1dff53dc4e04b39962fd85db131d6d"
server_key_version: "encryptionkey@0"
I20260925 09:17:17.636188 18760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "1c76d36dcf4749e5888ae8ab5e896e43"
format_stamp: "Formatted at 2026-09-25 09:17:17 on dist-test-slave-l6qh"
server_key: "270bc535692164dbb5798141f74570bc"
server_key_iv: "dd1dff53dc4e04b39962fd85db131d6d"
server_key_version: "encryptionkey@0"
I20260925 09:17:17.641611 18760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 09:17:17.644997 18775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:17.646283 18760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:17:17.646464 18760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "1c76d36dcf4749e5888ae8ab5e896e43"
format_stamp: "Formatted at 2026-09-25 09:17:17 on dist-test-slave-l6qh"
server_key: "270bc535692164dbb5798141f74570bc"
server_key_iv: "dd1dff53dc4e04b39962fd85db131d6d"
server_key_version: "encryptionkey@0"
I20260925 09:17:17.646680 18760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:17:17.676014 18760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:17:17.679761 18760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:17:17.680019 18760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:17:17.698294 18760 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:46573
I20260925 09:17:17.698302 18826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:46573 every 8 connection(s)
I20260925 09:17:17.699647 18760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:17:17.703218 18827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:17:17.708796 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 18760
I20260925 09:17:17.709028 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:17:17.709770 3149 external_mini_cluster.cc:1428] Setting key 0d21ef1f430b4ef19f53ab6bdd6f5a96
I20260925 09:17:17.713642 18827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Bootstrap starting.
I20260925 09:17:17.717263 18827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Neither blocks nor log segments found. Creating new log.
I20260925 09:17:17.718310 18827 log.cc:824] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Log is configured to *not* fsync() on all Append() calls
I20260925 09:17:17.721323 18827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: No bootstrap required, opened a new log
Sep 25 09:17:17 dist-test-slave-l6qh krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:17:17.730270 18827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } }
I20260925 09:17:17.730645 18827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:17:17.730779 18827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c76d36dcf4749e5888ae8ab5e896e43, State: Initialized, Role: FOLLOWER
I20260925 09:17:17.731344 18827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } }
I20260925 09:17:17.731595 18827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:17:17.731762 18827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:17:17.731967 18827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:17:17.733712 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:17.711866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.3.19.126:46573)
0925 09:17:17.712639 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:17.712651 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:17.712689 (+ 38us) server_negotiation.cc:407] Connection header received
0925 09:17:17.714400 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:17.714421 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:17.714765 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:17.715089 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:17.716956 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:17.718389 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:17.720147 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:17.720794 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:17.720928 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:17.725963 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:17.725971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:17.725986 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:17.726028 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:17.729144 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:17.729888 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:17.729896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:17.729912 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:17.730065 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:17.730447 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:17.730453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:17.730456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:17.731142 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:17.731466 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:17.731934 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 09:17:17.732369 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":374,"threads_started":1}
I20260925 09:17:17.734865 18827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } }
I20260925 09:17:17.735436 18827 leader_election.cc:302] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c76d36dcf4749e5888ae8ab5e896e43; no voters:
I20260925 09:17:17.736045 18827 leader_election.cc:288] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:17:17.736145 18832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:17:17.737147 18832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 1 LEADER]: Becoming Leader. State: Replica: 1c76d36dcf4749e5888ae8ab5e896e43, State: Running, Role: LEADER
I20260925 09:17:17.737819 18832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } }
I20260925 09:17:17.738518 18827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:17:17.740816 18833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } } }
I20260925 09:17:17.740895 18834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c76d36dcf4749e5888ae8ab5e896e43. Latest consensus state: current_term: 1 leader_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } } }
I20260925 09:17:17.741220 18833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: This master's current role is: LEADER
I20260925 09:17:17.741364 18834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: This master's current role is: LEADER
I20260925 09:17:17.742965 18837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:17:17.749645 18837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:17:17.759457 18837 catalog_manager.cc:1347] Generated new cluster ID: 273845f7479c4c5395cd989713e3bbfb
I20260925 09:17:17.759624 18837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:17:17.777369 18837 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:17:17.778563 18837 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:17:17.786829 18837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Generated new TSK 0
I20260925 09:17:17.787607 18837 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:17:17.857966 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46573
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:17:18.191033 18853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:17:18.191306 18853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:17:18.191399 18853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:17:18.198225 18853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:17:18.198323 18853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:17:18.198437 18853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:17:18.207472 18853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:17:18.208608 18853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:17:18.210008 18853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:17:18.222807 18858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:18.222834 18859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:18.222896 18861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:17:18.223619 18853 server_base.cc:1034] running on GCE node
I20260925 09:17:18.224247 18853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:17:18.225163 18853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:17:18.226411 18853 hybrid_clock.cc:648] HybridClock initialized: now 1790327838226376 us; error 31 us; skew 500 ppm
I20260925 09:17:18.226711 18853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:17:18 dist-test-slave-l6qh krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327838, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:17:18.229708 18853 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:17:18.231057 18853 webserver.cc:460] Webserver started at http://127.3.19.65:45013/ using document root <none> and password file <none>
I20260925 09:17:18.231724 18853 fs_manager.cc:359] Metadata directory not provided
I20260925 09:17:18.231865 18853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:17:18.232220 18853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:17:18.235160 18853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "16494a03f1a046f58757e4f7b029e0e3"
format_stamp: "Formatted at 2026-09-25 09:17:18 on dist-test-slave-l6qh"
server_key: "18cc30db1971e854254ca0b0671a405b"
server_key_iv: "f7e825d37fb328736bdb5955df671c93"
server_key_version: "encryptionkey@0"
I20260925 09:17:18.235845 18853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "16494a03f1a046f58757e4f7b029e0e3"
format_stamp: "Formatted at 2026-09-25 09:17:18 on dist-test-slave-l6qh"
server_key: "18cc30db1971e854254ca0b0671a405b"
server_key_iv: "f7e825d37fb328736bdb5955df671c93"
server_key_version: "encryptionkey@0"
I20260925 09:17:18.240918 18853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 09:17:18.244285 18868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:18.245702 18853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:17:18.245833 18853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "16494a03f1a046f58757e4f7b029e0e3"
format_stamp: "Formatted at 2026-09-25 09:17:18 on dist-test-slave-l6qh"
server_key: "18cc30db1971e854254ca0b0671a405b"
server_key_iv: "f7e825d37fb328736bdb5955df671c93"
server_key_version: "encryptionkey@0"
I20260925 09:17:18.245990 18853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:17:18.271610 18853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:17:18.275468 18853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:17:18.275697 18853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:17:18.276553 18853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:17:18.277872 18853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:17:18.277949 18853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:18.278028 18853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:17:18.278070 18853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:18.313366 18853 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:37555
I20260925 09:17:18.313455 18980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:37555 every 8 connection(s)
I20260925 09:17:18.314633 18853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:17:18.324046 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 18853
I20260925 09:17:18.324250 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:17:18.325026 3149 external_mini_cluster.cc:1428] Setting key 32e61af1335bc27e0f668a9a4d306a71
Sep 25 09:17:18 dist-test-slave-l6qh krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327838, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:17:18.335325 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:18.317129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:53109 (local address 127.3.19.126:46573)
0925 09:17:18.317617 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:18.317622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:18.318560 (+ 938us) server_negotiation.cc:407] Connection header received
0925 09:17:18.320376 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:18.320382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:18.320475 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:18.320595 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:18.322156 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:18.322899 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:18.325020 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:18.325529 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:18.325670 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:18.329656 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:18.329662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:18.329665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:18.329697 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:18.332313 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:18.333054 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:18.333058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:18.333061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:18.333174 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:18.333634 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:18.333640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:18.333643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:18.333923 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:18.334078 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:18.334747 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 09:17:18.334965 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":287,"threads_started":1}
I20260925 09:17:18.336241 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:18.317665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:53109)
0925 09:17:18.318396 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:18.318442 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:18.320110 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:18.320828 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:18.320836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:18.321419 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:17:18.321988 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:18.322004 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:18.323055 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:18.323062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:18.324873 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:18.324888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:18.325745 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:18.325749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:18.326008 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:18.326837 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:17:18.326866 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:17:18.329449 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:17:18.332454 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:18.332462 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:18.332479 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:18.332915 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:18.333327 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:18.333332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:18.333335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:18.333513 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:18.334048 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:17:18.334055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:17:18.334379 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 09:17:18.334631 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 09:17:18.335127 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":277,"threads_started":1}
I20260925 09:17:18.337993 18981 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46573
I20260925 09:17:18.338322 18981 heartbeater.cc:464] Registering TS with master...
I20260925 09:17:18.339208 18981 heartbeater.cc:511] Master 127.3.19.126:46573 requested a full tablet report, sending...
I20260925 09:17:18.341456 18792 ts_manager.cc:194] Registered new tserver with Master: 16494a03f1a046f58757e4f7b029e0e3 (127.3.19.65:37555)
I20260925 09:17:18.342896 18792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:53109
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:17:18.382922 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46573
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:17:18.714874 18987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:17:18.715216 18987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:17:18.715399 18987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:17:18.722817 18987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:17:18.722997 18987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:17:18.723191 18987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:17:18.733167 18987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:17:18.734546 18987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:17:18.735910 18987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:17:18.749724 18993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:18.750392 18995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:18.750415 18992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:17:18.750865 18987 server_base.cc:1034] running on GCE node
I20260925 09:17:18.751713 18987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:17:18.752765 18987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:17:18.754011 18987 hybrid_clock.cc:648] HybridClock initialized: now 1790327838753964 us; error 41 us; skew 500 ppm
I20260925 09:17:18.754360 18987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:17:18 dist-test-slave-l6qh krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327838, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:17:18.757711 18987 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:17:18.759196 18987 webserver.cc:460] Webserver started at http://127.3.19.66:35665/ using document root <none> and password file <none>
I20260925 09:17:18.759927 18987 fs_manager.cc:359] Metadata directory not provided
I20260925 09:17:18.760080 18987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:17:18.760411 18987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:17:18.763525 18987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "2196952eacd849eb8a0670f02dae647a"
format_stamp: "Formatted at 2026-09-25 09:17:18 on dist-test-slave-l6qh"
server_key: "59a5b4ad0359eaac4077c7de1d0ec529"
server_key_iv: "4ab09387d2a0968a21b22b239e931260"
server_key_version: "encryptionkey@0"
I20260925 09:17:18.764299 18987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "2196952eacd849eb8a0670f02dae647a"
format_stamp: "Formatted at 2026-09-25 09:17:18 on dist-test-slave-l6qh"
server_key: "59a5b4ad0359eaac4077c7de1d0ec529"
server_key_iv: "4ab09387d2a0968a21b22b239e931260"
server_key_version: "encryptionkey@0"
I20260925 09:17:18.769672 18987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 09:17:18.773185 19002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:18.774626 18987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:17:18.774802 18987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "2196952eacd849eb8a0670f02dae647a"
format_stamp: "Formatted at 2026-09-25 09:17:18 on dist-test-slave-l6qh"
server_key: "59a5b4ad0359eaac4077c7de1d0ec529"
server_key_iv: "4ab09387d2a0968a21b22b239e931260"
server_key_version: "encryptionkey@0"
I20260925 09:17:18.775014 18987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:17:18.800125 18987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:17:18.804811 18987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:17:18.805125 18987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:17:18.806203 18987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:17:18.807546 18987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:17:18.807653 18987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:18.807816 18987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:17:18.807890 18987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:18.843608 18987 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:37315
I20260925 09:17:18.843704 19114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:37315 every 8 connection(s)
I20260925 09:17:18.844949 18987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:17:18.849666 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 18987
I20260925 09:17:18.849872 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:17:18.850378 3149 external_mini_cluster.cc:1428] Setting key 738f9e872973c0866a5dedf43724ef03
Sep 25 09:17:18 dist-test-slave-l6qh krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327838, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:17:18.866345 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:18.847673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:40237 (local address 127.3.19.126:46573)
0925 09:17:18.848119 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:18.848123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:18.849316 (+ 1193us) server_negotiation.cc:407] Connection header received
0925 09:17:18.851901 (+ 2585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:18.851907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:18.852004 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:18.852262 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:18.853830 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:18.854570 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:18.856154 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:18.856590 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:18.856791 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:18.860547 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:18.860552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:18.860555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:18.860587 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:18.863276 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:18.863992 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:18.864001 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:18.864004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:18.864127 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:18.864557 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:18.864566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:18.864568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:18.864830 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:18.864986 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:18.865733 (+ 747us) server_negotiation.cc:299] Negotiation successful
0925 09:17:18.865956 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":226,"threads_started":1}
I20260925 09:17:18.867357 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:18.848279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:40237)
0925 09:17:18.849047 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:18.849169 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:18.851618 (+ 2449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:18.852402 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:18.852415 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:18.852982 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:17:18.853656 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:18.853678 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:18.854733 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:18.854738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:18.856003 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:18.856014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:18.856862 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:18.856869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:18.857118 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:18.857911 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:17:18.857933 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:17:18.860320 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:17:18.863415 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:18.863424 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:18.863442 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:18.863863 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:18.864244 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:18.864249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:18.864253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:18.864436 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:18.864967 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:17:18.864974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:17:18.865370 (+ 396us) client_negotiation.cc:769] Sending connection context
0925 09:17:18.865633 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 09:17:18.866190 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":282,"threads_started":1}
I20260925 09:17:18.868892 19115 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46573
I20260925 09:17:18.869282 19115 heartbeater.cc:464] Registering TS with master...
I20260925 09:17:18.870011 19115 heartbeater.cc:511] Master 127.3.19.126:46573 requested a full tablet report, sending...
I20260925 09:17:18.871583 18792 ts_manager.cc:194] Registered new tserver with Master: 2196952eacd849eb8a0670f02dae647a (127.3.19.66:37315)
I20260925 09:17:18.872924 18792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:40237
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:17:18.909489 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46573
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:17:19.245733 19121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:17:19.246029 19121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:17:19.246121 19121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:17:19.253697 19121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:17:19.253824 19121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:17:19.253938 19121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:17:19.264163 19121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:46573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:17:19.265622 19121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:17:19.266932 19121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:17:19.280485 19126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:19.280359 19127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:19.280345 19129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:17:19.281147 19121 server_base.cc:1034] running on GCE node
I20260925 09:17:19.281839 19121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:17:19.282876 19121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:17:19.284137 19121 hybrid_clock.cc:648] HybridClock initialized: now 1790327839284101 us; error 33 us; skew 500 ppm
I20260925 09:17:19.284626 19121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:17:19 dist-test-slave-l6qh krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327839, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:17:19.288305 19121 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:17:19.289978 19121 webserver.cc:460] Webserver started at http://127.3.19.67:46051/ using document root <none> and password file <none>
I20260925 09:17:19.290691 19121 fs_manager.cc:359] Metadata directory not provided
I20260925 09:17:19.290804 19121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:17:19.291172 19121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:17:19.294304 19121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "696fe6f9a0eb4ab7b9845853f1903b3e"
format_stamp: "Formatted at 2026-09-25 09:17:19 on dist-test-slave-l6qh"
server_key: "dc77e433ae6a48cd2075028676990ae5"
server_key_iv: "62deb258ab24e48f4e4e8c62b507f6f9"
server_key_version: "encryptionkey@0"
I20260925 09:17:19.295095 19121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "696fe6f9a0eb4ab7b9845853f1903b3e"
format_stamp: "Formatted at 2026-09-25 09:17:19 on dist-test-slave-l6qh"
server_key: "dc77e433ae6a48cd2075028676990ae5"
server_key_iv: "62deb258ab24e48f4e4e8c62b507f6f9"
server_key_version: "encryptionkey@0"
I20260925 09:17:19.300483 19121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 09:17:19.304140 19136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:19.305562 19121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:17:19.305737 19121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "696fe6f9a0eb4ab7b9845853f1903b3e"
format_stamp: "Formatted at 2026-09-25 09:17:19 on dist-test-slave-l6qh"
server_key: "dc77e433ae6a48cd2075028676990ae5"
server_key_iv: "62deb258ab24e48f4e4e8c62b507f6f9"
server_key_version: "encryptionkey@0"
I20260925 09:17:19.305943 19121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:17:19.322911 19121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:17:19.327400 19121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:17:19.327718 19121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:17:19.328920 19121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:17:19.330389 19121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:17:19.330498 19121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:19.330657 19121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:17:19.330734 19121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:19.346643 18981 heartbeater.cc:503] Master 127.3.19.126:46573 was elected leader, sending a full tablet report...
I20260925 09:17:19.365985 19121 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:37195
I20260925 09:17:19.366031 19248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:37195 every 8 connection(s)
I20260925 09:17:19.367499 19121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:17:19.374830 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 19121
I20260925 09:17:19.375054 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:17:19.375540 3149 external_mini_cluster.cc:1428] Setting key f65dce19844062e70a5f28ac5cb320cf
Sep 25 09:17:19 dist-test-slave-l6qh krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327839, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:17:19.388641 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.370381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:51769 (local address 127.3.19.126:46573)
0925 09:17:19.370836 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:19.370840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:19.371888 (+ 1048us) server_negotiation.cc:407] Connection header received
0925 09:17:19.373883 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:19.373889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:19.374000 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:19.374122 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:19.375925 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.376865 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.378696 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.379219 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.379322 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.382963 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:19.382968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:19.382972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:19.383008 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:19.385705 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:19.386446 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:19.386451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:19.386453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:19.386579 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:19.387035 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:19.387038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:19.387040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:19.387331 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:19.387451 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:19.388167 (+ 716us) server_negotiation.cc:299] Negotiation successful
0925 09:17:19.388356 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":215,"threads_started":1}
I20260925 09:17:19.389850 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.370979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:51769)
0925 09:17:19.371699 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:19.371744 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:19.373573 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:19.374388 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:19.374397 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:19.374944 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:17:19.375724 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:19.375745 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.377034 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.377038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:19.378538 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:19.378555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.379454 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.379459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:19.379670 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.380393 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:17:19.380416 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:17:19.382739 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:17:19.385861 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:19.385870 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:19.385884 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:19.386315 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:19.386671 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:19.386674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:19.386676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:19.386922 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:19.387437 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:17:19.387443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:17:19.387785 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 09:17:19.388063 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 09:17:19.388495 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":297,"threads_started":1}
I20260925 09:17:19.391372 19249 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46573
I20260925 09:17:19.391732 19249 heartbeater.cc:464] Registering TS with master...
I20260925 09:17:19.392504 19249 heartbeater.cc:511] Master 127.3.19.126:46573 requested a full tablet report, sending...
I20260925 09:17:19.394104 18791 ts_manager.cc:194] Registered new tserver with Master: 696fe6f9a0eb4ab7b9845853f1903b3e (127.3.19.67:37195)
I20260925 09:17:19.394990 18791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:51769
I20260925 09:17:19.404114 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:17:19.422820 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.408891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.3.19.126:46573)
0925 09:17:19.409076 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:19.409081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:19.409460 (+ 379us) server_negotiation.cc:407] Connection header received
0925 09:17:19.409683 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:19.409689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:19.409808 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:19.409919 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:19.411569 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.412324 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.413821 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.414272 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.414423 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.417107 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:19.417112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:19.417115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:19.417150 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:19.419937 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:19.420663 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:19.420670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:19.420674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:19.420811 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:19.421221 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:19.421255 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:19.421259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:19.421615 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:19.421897 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:19.422235 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 09:17:19.422482 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 09:17:19.427388 18791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:17:19.434616 18791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:17:19.464013 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.447391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.3.19.66:37315)
0925 09:17:19.449501 (+ 2110us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:19.449513 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:19.449544 (+ 31us) server_negotiation.cc:407] Connection header received
0925 09:17:19.449654 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:19.449673 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:19.449952 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:19.450393 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:17:19.452326 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.454466 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.461335 (+ 6869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.462563 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.462726 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.462957 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:19.463448 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 09:17:19.463639 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1941,"thread_start_us":1773,"threads_started":1}
I20260925 09:17:19.464103 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.447199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:37315 (local address 127.0.0.1:52492)
0925 09:17:19.448087 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:19.448129 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:19.448321 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:19.450789 (+ 2468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:19.450797 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:19.450836 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:17:19.452041 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:19.452059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.454820 (+ 2761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.454829 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:19.459114 (+ 4285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:19.459138 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.462783 (+ 3645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.462790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:19.463054 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.463196 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 09:17:19.463361 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 09:17:19.463582 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":115,"thread_start_us":1514,"threads_started":1}
I20260925 09:17:19.464635 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.447294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:37195 (local address 127.0.0.1:48892)
0925 09:17:19.449035 (+ 1741us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:19.449108 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:19.449273 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:19.450742 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:19.450750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:19.450785 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:17:19.451729 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:19.451780 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.455177 (+ 3397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.455186 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:19.458996 (+ 3810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:19.459020 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.463174 (+ 4154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.463180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:19.463367 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.463397 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 09:17:19.463473 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 09:17:19.463582 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1578,"mutex_wait_us":167,"thread_start_us":726,"threads_started":1}
I20260925 09:17:19.464666 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.447885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.3.19.67:37195)
0925 09:17:19.449857 (+ 1972us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:19.449865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:19.449893 (+ 28us) server_negotiation.cc:407] Connection header received
0925 09:17:19.449982 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:19.449991 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:19.450231 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:19.450402 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:17:19.451959 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.454260 (+ 2301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.459246 (+ 4986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.461288 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.461398 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.461656 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:19.464115 (+ 2459us) server_negotiation.cc:299] Negotiation successful
0925 09:17:19.464326 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1821,"thread_start_us":1685,"threads_started":1}
I20260925 09:17:19.468575 19184 tablet_service.cc:1467] Processing CreateTablet for tablet 0568f88e277a4f19890b56339628b05f (DEFAULT_TABLE table=test-table [id=1aa54fa08a364aa3922baf3947b8706d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:17:19.468578 19050 tablet_service.cc:1467] Processing CreateTablet for tablet 0568f88e277a4f19890b56339628b05f (DEFAULT_TABLE table=test-table [id=1aa54fa08a364aa3922baf3947b8706d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:17:19.469858 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.450272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:37555 (local address 127.0.0.1:58294)
0925 09:17:19.461395 (+ 11123us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:19.461422 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:19.461581 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:19.462796 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:19.462800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:19.462826 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:17:19.463379 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:19.463393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.466281 (+ 2888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.466290 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:19.467603 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:19.467615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.468941 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.468947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:19.469120 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.469156 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 09:17:19.469331 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 09:17:19.469472 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10886,"mutex_wait_us":23,"thread_start_us":10668,"threads_started":1}
I20260925 09:17:19.470098 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.450449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.3.19.65:37555)
0925 09:17:19.462222 (+ 11773us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:19.462238 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:19.462265 (+ 27us) server_negotiation.cc:407] Connection header received
0925 09:17:19.462354 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:19.462366 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:19.462578 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:19.462831 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:17:19.463580 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.466142 (+ 2562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.467781 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.468743 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.468896 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.469133 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:19.469479 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 09:17:19.469656 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11607,"thread_start_us":11437,"threads_started":1}
I20260925 09:17:19.470191 19184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0568f88e277a4f19890b56339628b05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:17:19.470192 19050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0568f88e277a4f19890b56339628b05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:17:19.473635 18916 tablet_service.cc:1467] Processing CreateTablet for tablet 0568f88e277a4f19890b56339628b05f (DEFAULT_TABLE table=test-table [id=1aa54fa08a364aa3922baf3947b8706d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:17:19.475132 18916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0568f88e277a4f19890b56339628b05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:17:19.486102 19269 tablet_bootstrap.cc:492] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e: Bootstrap starting.
I20260925 09:17:19.487054 19268 tablet_bootstrap.cc:492] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a: Bootstrap starting.
I20260925 09:17:19.487998 19270 tablet_bootstrap.cc:492] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3: Bootstrap starting.
I20260925 09:17:19.491482 19269 tablet_bootstrap.cc:654] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e: Neither blocks nor log segments found. Creating new log.
I20260925 09:17:19.492321 19270 tablet_bootstrap.cc:654] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3: Neither blocks nor log segments found. Creating new log.
I20260925 09:17:19.492322 19268 tablet_bootstrap.cc:654] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a: Neither blocks nor log segments found. Creating new log.
I20260925 09:17:19.492640 19269 log.cc:824] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e: Log is configured to *not* fsync() on all Append() calls
I20260925 09:17:19.493726 19268 log.cc:824] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a: Log is configured to *not* fsync() on all Append() calls
I20260925 09:17:19.493723 19270 log.cc:824] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3: Log is configured to *not* fsync() on all Append() calls
I20260925 09:17:19.496033 19269 tablet_bootstrap.cc:492] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e: No bootstrap required, opened a new log
I20260925 09:17:19.496317 19269 ts_tablet_manager.cc:1397] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260925 09:17:19.496907 19270 tablet_bootstrap.cc:492] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3: No bootstrap required, opened a new log
I20260925 09:17:19.497270 19270 ts_tablet_manager.cc:1397] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 09:17:19.497872 19268 tablet_bootstrap.cc:492] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a: No bootstrap required, opened a new log
I20260925 09:17:19.498167 19268 ts_tablet_manager.cc:1397] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
I20260925 09:17:19.504688 19269 raft_consensus.cc:348] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:19.505157 19269 raft_consensus.cc:374] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:17:19.505300 19269 raft_consensus.cc:729] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696fe6f9a0eb4ab7b9845853f1903b3e, State: Initialized, Role: FOLLOWER
I20260925 09:17:19.506099 19269 consensus_queue.cc:260] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:19.507421 19249 heartbeater.cc:503] Master 127.3.19.126:46573 was elected leader, sending a full tablet report...
I20260925 09:17:19.509084 19270 raft_consensus.cc:348] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:19.509661 19270 raft_consensus.cc:374] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:17:19.509799 19270 raft_consensus.cc:729] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16494a03f1a046f58757e4f7b029e0e3, State: Initialized, Role: FOLLOWER
I20260925 09:17:19.510299 19268 raft_consensus.cc:348] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:19.510583 19270 consensus_queue.cc:260] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:19.510805 19268 raft_consensus.cc:374] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:17:19.510922 19268 raft_consensus.cc:729] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2196952eacd849eb8a0670f02dae647a, State: Initialized, Role: FOLLOWER
I20260925 09:17:19.511725 19268 consensus_queue.cc:260] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:19.513135 19269 ts_tablet_manager.cc:1428] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e: Time spent starting tablet: real 0.017s user 0.013s sys 0.001s
I20260925 09:17:19.514488 19270 ts_tablet_manager.cc:1428] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3: Time spent starting tablet: real 0.017s user 0.005s sys 0.009s
I20260925 09:17:19.516503 19115 heartbeater.cc:503] Master 127.3.19.126:46573 was elected leader, sending a full tablet report...
I20260925 09:17:19.517360 19268 ts_tablet_manager.cc:1428] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a: Time spent starting tablet: real 0.019s user 0.017s sys 0.002s
W20260925 09:17:19.567258 18982 tablet.cc:2369] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:17:19.597340 19116 tablet.cc:2369] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:17:19.619401 19250 tablet.cc:2369] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:17:19.640745 19274 raft_consensus.cc:482] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:17:19.641072 19274 raft_consensus.cc:504] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:19.642820 19274 leader_election.cc:288] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2196952eacd849eb8a0670f02dae647a (127.3.19.66:37315), 16494a03f1a046f58757e4f7b029e0e3 (127.3.19.65:37555)
I20260925 09:17:19.649369 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.643108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:37555 (local address 127.3.19.67:32989)
0925 09:17:19.643400 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:19.643423 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:19.643569 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:19.643986 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:19.643989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:19.644016 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:17:19.644225 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:19.644236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.645537 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.645543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:19.647022 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:19.647035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.648504 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.648510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:19.648721 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.648758 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 09:17:19.648871 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 09:17:19.649020 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260925 09:17:19.649708 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.643335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:32989 (local address 127.3.19.65:37555)
0925 09:17:19.643520 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:19.643528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:19.643549 (+ 21us) server_negotiation.cc:407] Connection header received
0925 09:17:19.643747 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:19.643755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:19.643843 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:19.643954 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:17:19.644364 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.645402 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.647174 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.648343 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.648487 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.648604 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:19.648963 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 09:17:19.649186 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 09:17:19.650122 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.643264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:53531 (local address 127.3.19.66:37315)
0925 09:17:19.643474 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:19.643480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:19.644139 (+ 659us) server_negotiation.cc:407] Connection header received
0925 09:17:19.644264 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:19.644272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:19.644378 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:19.644451 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:17:19.645038 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.645977 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.647900 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.648894 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.649006 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.649088 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:19.649644 (+ 556us) server_negotiation.cc:299] Negotiation successful
0925 09:17:19.649824 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 09:17:19.650888 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.643180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:37315 (local address 127.3.19.67:53531)
0925 09:17:19.644019 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:19.644037 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:19.644159 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:19.644496 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:19.644502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:19.644526 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:17:19.644890 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:19.644902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.646123 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.646126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:19.647698 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:19.647712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.649250 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.649256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:19.649449 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.649484 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 09:17:19.650310 (+ 826us) client_negotiation.cc:240] Negotiation successful
0925 09:17:19.650517 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":479,"threads_started":1}
I20260925 09:17:19.651098 18936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0568f88e277a4f19890b56339628b05f" candidate_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16494a03f1a046f58757e4f7b029e0e3" is_pre_election: true
I20260925 09:17:19.651741 18936 raft_consensus.cc:2454] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696fe6f9a0eb4ab7b9845853f1903b3e in term 0.
I20260925 09:17:19.652112 19070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0568f88e277a4f19890b56339628b05f" candidate_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2196952eacd849eb8a0670f02dae647a" is_pre_election: true
I20260925 09:17:19.652684 19138 leader_election.cc:302] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16494a03f1a046f58757e4f7b029e0e3, 696fe6f9a0eb4ab7b9845853f1903b3e; no voters:
I20260925 09:17:19.652740 19070 raft_consensus.cc:2454] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696fe6f9a0eb4ab7b9845853f1903b3e in term 0.
I20260925 09:17:19.653162 19274 raft_consensus.cc:2781] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:17:19.653345 19274 raft_consensus.cc:482] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:17:19.653462 19274 raft_consensus.cc:3037] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:17:19.656644 19274 raft_consensus.cc:504] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:19.657500 19274 leader_election.cc:288] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [CANDIDATE]: Term 1 election: Requested vote from peers 2196952eacd849eb8a0670f02dae647a (127.3.19.66:37315), 16494a03f1a046f58757e4f7b029e0e3 (127.3.19.65:37555)
I20260925 09:17:19.658004 19070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0568f88e277a4f19890b56339628b05f" candidate_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2196952eacd849eb8a0670f02dae647a"
I20260925 09:17:19.658257 19070 raft_consensus.cc:3037] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:17:19.658581 18936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0568f88e277a4f19890b56339628b05f" candidate_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16494a03f1a046f58757e4f7b029e0e3"
I20260925 09:17:19.658807 18936 raft_consensus.cc:3037] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:17:19.660588 19070 raft_consensus.cc:2454] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696fe6f9a0eb4ab7b9845853f1903b3e in term 1.
I20260925 09:17:19.660949 18936 raft_consensus.cc:2454] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696fe6f9a0eb4ab7b9845853f1903b3e in term 1.
I20260925 09:17:19.660962 19139 leader_election.cc:302] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2196952eacd849eb8a0670f02dae647a, 696fe6f9a0eb4ab7b9845853f1903b3e; no voters:
I20260925 09:17:19.661298 19274 raft_consensus.cc:2781] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:17:19.661746 19274 raft_consensus.cc:686] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [term 1 LEADER]: Becoming Leader. State: Replica: 696fe6f9a0eb4ab7b9845853f1903b3e, State: Running, Role: LEADER
I20260925 09:17:19.662257 19274 consensus_queue.cc:237] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:19.666730 18792 catalog_manager.cc:5495] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 696fe6f9a0eb4ab7b9845853f1903b3e (127.3.19.67). New cstate: current_term: 1 leader_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } health_report { overall_health: UNKNOWN } } }
I20260925 09:17:19.699035 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.693776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.3.19.67:37195)
0925 09:17:19.694020 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:19.694024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:19.694041 (+ 17us) server_negotiation.cc:407] Connection header received
0925 09:17:19.694216 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:19.694220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:19.694298 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:19.694423 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:17:19.694737 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.695459 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.696887 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.697335 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.697428 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.697950 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:17:19.698460 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:17:19.698623 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:19.698691 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 09:17:19.698777 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 09:17:19.708240 19070 raft_consensus.cc:1261] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a [term 1 FOLLOWER]: Refusing update from remote peer 696fe6f9a0eb4ab7b9845853f1903b3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:17:19.708485 18936 raft_consensus.cc:1261] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3 [term 1 FOLLOWER]: Refusing update from remote peer 696fe6f9a0eb4ab7b9845853f1903b3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:17:19.709285 19274 consensus_queue.cc:1035] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:17:19.709581 19278 consensus_queue.cc:1035] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:17:19.723800 19281 mvcc.cc:205] Tried to move back new op lower bound from 7333182831434928128 to 7333182831260971008. Current Snapshot: MvccSnapshot[applied={T|T < 7333182831434928128}]
I20260925 09:17:19.737449 18791 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:17:19.737705 18791 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:17:19.741286 18791 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Sending DeleteTablet for 3 replicas of tablet 0568f88e277a4f19890b56339628b05f
I20260925 09:17:19.742400 19050 tablet_service.cc:1514] Processing DeleteTablet for tablet 0568f88e277a4f19890b56339628b05f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:17:19 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:52492
I20260925 09:17:19.742429 19184 tablet_service.cc:1514] Processing DeleteTablet for tablet 0568f88e277a4f19890b56339628b05f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:17:19 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:48892
I20260925 09:17:19.742455 18916 tablet_service.cc:1514] Processing DeleteTablet for tablet 0568f88e277a4f19890b56339628b05f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:17:19 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:58294
I20260925 09:17:19.743069 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 18853
I20260925 09:17:19.757825 19291 tablet_replica.cc:330] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3: stopping tablet replica
I20260925 09:17:19.758399 19291 raft_consensus.cc:2229] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:17:19.759004 19291 raft_consensus.cc:2258] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:17:19.761785 19292 tablet_replica.cc:330] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a: stopping tablet replica
I20260925 09:17:19.762255 19292 raft_consensus.cc:2229] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:17:19.762761 19292 raft_consensus.cc:2258] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:17:19.764883 19291 ts_tablet_manager.cc:1905] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:17:19.765496 19293 tablet_replica.cc:330] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e: stopping tablet replica
I20260925 09:17:19.765868 19293 raft_consensus.cc:2229] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:17:19.766315 19292 ts_tablet_manager.cc:1905] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:17:19.766497 19293 raft_consensus.cc:2258] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:17:19.768432 19293 ts_tablet_manager.cc:1905] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:17:19.773684 19291 ts_tablet_manager.cc:1918] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:17:19.773850 19291 log.cc:1196] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/0568f88e277a4f19890b56339628b05f
I20260925 09:17:19.774155 19292 ts_tablet_manager.cc:1918] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:17:19.774307 19292 log.cc:1196] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/0568f88e277a4f19890b56339628b05f
I20260925 09:17:19.774402 19291 ts_tablet_manager.cc:1939] T 0568f88e277a4f19890b56339628b05f P 16494a03f1a046f58757e4f7b029e0e3: Deleting consensus metadata
I20260925 09:17:19.774828 19292 ts_tablet_manager.cc:1939] T 0568f88e277a4f19890b56339628b05f P 2196952eacd849eb8a0670f02dae647a: Deleting consensus metadata
I20260925 09:17:19.776294 18777 catalog_manager.cc:4841] TS 16494a03f1a046f58757e4f7b029e0e3 (127.3.19.65:37555): tablet 0568f88e277a4f19890b56339628b05f (table test-table [id=1aa54fa08a364aa3922baf3947b8706d]) successfully deleted
I20260925 09:17:19.776403 18778 catalog_manager.cc:4841] TS 2196952eacd849eb8a0670f02dae647a (127.3.19.66:37315): tablet 0568f88e277a4f19890b56339628b05f (table test-table [id=1aa54fa08a364aa3922baf3947b8706d]) successfully deleted
I20260925 09:17:19.777350 19293 ts_tablet_manager.cc:1918] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:17:19.777527 19293 log.cc:1196] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/0568f88e277a4f19890b56339628b05f
I20260925 09:17:19.777997 19293 ts_tablet_manager.cc:1939] T 0568f88e277a4f19890b56339628b05f P 696fe6f9a0eb4ab7b9845853f1903b3e: Deleting consensus metadata
Sep 25 09:17:19 dist-test-slave-l6qh krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:17:19.779539 18776 catalog_manager.cc:4841] TS 696fe6f9a0eb4ab7b9845853f1903b3e (127.3.19.67:37195): tablet 0568f88e277a4f19890b56339628b05f (table test-table [id=1aa54fa08a364aa3922baf3947b8706d]) successfully deleted
I20260925 09:17:19.786535 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.744614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.3.19.65:37555)
0925 09:17:19.744819 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:19.744827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:19.754067 (+ 9240us) server_negotiation.cc:407] Connection header received
0925 09:17:19.754570 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:19.754580 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:19.754690 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:19.754777 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:19.759507 (+ 4730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.761007 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.764107 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.764959 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.765088 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.780706 (+ 15618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:19.780718 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:19.780732 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:19.780772 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:19.783525 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:19.784305 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:19.784312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:19.784314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:19.784450 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:19.784866 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:19.784871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:19.784876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:19.785306 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:19.785609 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:19.785939 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 09:17:19.786160 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 09:17:19.787322 18976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:17:19.875435 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 18853
I20260925 09:17:19.893858 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 18987
Sep 25 09:17:19 dist-test-slave-l6qh krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:17:19.909726 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:19.894716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.3.19.66:37315)
0925 09:17:19.894932 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:19.894939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:19.894961 (+ 22us) server_negotiation.cc:407] Connection header received
0925 09:17:19.895086 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:19.895095 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:19.895167 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:19.895282 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:19.896558 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.897416 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.898830 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:19.899217 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:19.899355 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:19.903809 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:19.903822 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:19.903837 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:19.903869 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:19.906845 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:19.907583 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:19.907592 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:19.907595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:19.907710 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:19.908078 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:19.908084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:19.908088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:19.908460 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:19.908747 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:19.909093 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 09:17:19.909338 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 09:17:19.910514 19110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:17:20.000283 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 18987
I20260925 09:17:20.017935 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 19121
Sep 25 09:17:20 dist-test-slave-l6qh krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:17:20.034343 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:20.018738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48922 (local address 127.3.19.67:37195)
0925 09:17:20.018918 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:20.018925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:20.018952 (+ 27us) server_negotiation.cc:407] Connection header received
0925 09:17:20.019152 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:20.019158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:20.019233 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:20.019351 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:20.020659 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:20.021543 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:20.022817 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:20.023274 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:20.023409 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:20.028082 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:20.028093 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:20.028107 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:20.028151 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:20.031390 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:20.032054 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:20.032068 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:20.032070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:20.032238 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:20.032637 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:20.032644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:20.032646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:20.033002 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:20.033553 (+ 551us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:20.033726 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 09:17:20.033961 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 09:17:20.035060 19244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:17:20.153900 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 19121
I20260925 09:17:20.173700 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 18760
I20260925 09:17:20.174785 18822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:17:20.282640 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 18760
I20260925 09:17:40.304898 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46573
--webserver_interface=127.3.19.126
--webserver_port=37393
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:46573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:17:40.638976 19299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:17:40.639241 19299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:17:40.639431 19299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:17:40.646540 19299 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 09:17:40.646651 19299 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 09:17:40.646732 19299 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 09:17:40.646822 19299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:17:40.646894 19299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:17:40.647001 19299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:17:40.647051 19299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:17:40.647142 19299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:17:40.656980 19299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:46573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=37393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:17:40.658320 19299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:17:40.659669 19299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:17:40.683970 19306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:40.683986 19305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:40.684267 19308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:17:40.684893 19299 server_base.cc:1034] running on GCE node
I20260925 09:17:40.685782 19299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:17:40.687301 19299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:17:40.688531 19299 hybrid_clock.cc:648] HybridClock initialized: now 1790327860688446 us; error 56 us; skew 500 ppm
I20260925 09:17:40.688889 19299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:17:40 dist-test-slave-l6qh krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327860, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:17:40.696161 19299 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:17:40.697999 19299 webserver.cc:460] Webserver started at http://127.3.19.126:37393/ using document root <none> and password file <none>
I20260925 09:17:40.698828 19299 fs_manager.cc:359] Metadata directory not provided
I20260925 09:17:40.698973 19299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:17:40.705021 19299 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 09:17:40.708178 19315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:40.709728 19299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:17:40.709916 19299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "1c76d36dcf4749e5888ae8ab5e896e43"
format_stamp: "Formatted at 2026-09-25 09:17:17 on dist-test-slave-l6qh"
server_key: "270bc535692164dbb5798141f74570bc"
server_key_iv: "dd1dff53dc4e04b39962fd85db131d6d"
server_key_version: "encryptionkey@0"
I20260925 09:17:40.710988 19299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:17:40.750669 19299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:17:40.754865 19299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:17:40.755182 19299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:17:40.774773 19299 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:46573
I20260925 09:17:40.774811 19366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:46573 every 8 connection(s)
I20260925 09:17:40.776237 19299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:17:40.779683 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 19299
I20260925 09:17:40.780719 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:37555
--local_ip_for_outbound_sockets=127.3.19.65
--tserver_master_addrs=127.3.19.126:46573
--webserver_port=45013
--webserver_interface=127.3.19.65
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:17:40.785188 19367 sys_catalog.cc:263] Verifying existing consensus state
I20260925 09:17:40.790565 19367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Bootstrap starting.
I20260925 09:17:40.810470 19367 log.cc:824] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Log is configured to *not* fsync() on all Append() calls
I20260925 09:17:40.822707 19367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 09:17:40.823302 19367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Bootstrap complete.
I20260925 09:17:40.832825 19367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } }
I20260925 09:17:40.833738 19367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c76d36dcf4749e5888ae8ab5e896e43, State: Initialized, Role: FOLLOWER
I20260925 09:17:40.834359 19367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } }
I20260925 09:17:40.834609 19367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:17:40.834744 19367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:17:40.834930 19367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 09:17:40.838573 19367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } }
I20260925 09:17:40.839087 19367 leader_election.cc:302] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c76d36dcf4749e5888ae8ab5e896e43; no voters:
I20260925 09:17:40.839769 19367 leader_election.cc:288] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 09:17:40.839897 19371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 09:17:40.841190 19371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 2 LEADER]: Becoming Leader. State: Replica: 1c76d36dcf4749e5888ae8ab5e896e43, State: Running, Role: LEADER
I20260925 09:17:40.841810 19371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } }
I20260925 09:17:40.842092 19367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:17:40.844388 19372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } } }
I20260925 09:17:40.844835 19372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: This master's current role is: LEADER
I20260925 09:17:40.848616 19373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c76d36dcf4749e5888ae8ab5e896e43. Latest consensus state: current_term: 2 leader_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } } }
I20260925 09:17:40.848953 19373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: This master's current role is: LEADER
I20260925 09:17:40.850854 19377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:17:40.860430 19377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:17:40.862313 19377 catalog_manager.cc:1259] Loaded cluster ID: 273845f7479c4c5395cd989713e3bbfb
I20260925 09:17:40.862421 19377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:17:40.866513 19377 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:17:40.868289 19377 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Loaded TSK: 0
I20260925 09:17:40.899570 19377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Generated new TSK 1
I20260925 09:17:40.899993 19377 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 09:17:41.142035 19369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:17:41.142289 19369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:17:41.142437 19369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:17:41.149583 19369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:17:41.149682 19369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:17:41.149871 19369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:17:41.159473 19369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:37555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=45013
--tserver_master_addrs=127.3.19.126:46573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:17:41.160730 19369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:17:41.162076 19369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:17:41.176962 19394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:41.177520 19397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:41.177511 19395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:17:41.178041 19369 server_base.cc:1034] running on GCE node
I20260925 09:17:41.178674 19369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:17:41.179647 19369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:17:41.180862 19369 hybrid_clock.cc:648] HybridClock initialized: now 1790327861180814 us; error 43 us; skew 500 ppm
I20260925 09:17:41.181186 19369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:17:41 dist-test-slave-l6qh krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327861, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:17:41.184836 19369 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:17:41.186555 19369 webserver.cc:460] Webserver started at http://127.3.19.65:45013/ using document root <none> and password file <none>
I20260925 09:17:41.187258 19369 fs_manager.cc:359] Metadata directory not provided
I20260925 09:17:41.187412 19369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:17:41.193408 19369 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 09:17:41.196409 19404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:41.197871 19369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:17:41.198082 19369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "16494a03f1a046f58757e4f7b029e0e3"
format_stamp: "Formatted at 2026-09-25 09:17:18 on dist-test-slave-l6qh"
server_key: "18cc30db1971e854254ca0b0671a405b"
server_key_iv: "f7e825d37fb328736bdb5955df671c93"
server_key_version: "encryptionkey@0"
I20260925 09:17:41.199082 19369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:17:41.218585 19369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:17:41.223999 19369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:17:41.224344 19369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:17:41.225627 19369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:17:41.227428 19369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:17:41.227548 19369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:41.228214 19369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:17:41.228309 19369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:41.266492 19369 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:37555
I20260925 09:17:41.266536 19516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:37555 every 8 connection(s)
I20260925 09:17:41.268518 19369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:17:41.277724 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 19369
I20260925 09:17:41.278393 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:37315
--local_ip_for_outbound_sockets=127.3.19.66
--tserver_master_addrs=127.3.19.126:46573
--webserver_port=35665
--webserver_interface=127.3.19.66
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 09:17:41 dist-test-slave-l6qh krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327861, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:17:41.295009 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:41.272066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:45465)
0925 09:17:41.273004 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:41.273057 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:41.274874 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:41.276117 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:41.276125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:41.276588 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:17:41.277172 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:41.277202 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:41.279139 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:41.279148 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:41.281443 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:41.281461 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:41.282486 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:41.282496 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:41.282859 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:41.283583 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:17:41.283616 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:17:41.286872 (+ 3256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:17:41.290587 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:41.290597 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:41.290619 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:41.291041 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:41.291515 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:41.291520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:41.291523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:41.291702 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:41.292461 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:17:41.292470 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:17:41.292823 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 09:17:41.293094 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 09:17:41.293674 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":388,"threads_started":1}
I20260925 09:17:41.295009 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:41.271768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:45465 (local address 127.3.19.126:46573)
0925 09:17:41.272507 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:41.272526 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:41.273178 (+ 652us) server_negotiation.cc:407] Connection header received
0925 09:17:41.275338 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:41.275351 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:41.275679 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:41.275956 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:41.277438 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:41.278963 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:41.281586 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:41.282252 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:41.282380 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:41.287202 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:41.287211 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:41.287227 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:41.287269 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:41.290408 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:41.291198 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:41.291207 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:41.291229 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:41.291394 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:41.291835 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:41.291840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:41.291844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:41.292319 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:41.292529 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:41.293348 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 09:17:41.293729 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":304,"threads_started":1}
I20260925 09:17:41.298136 19517 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46573
I20260925 09:17:41.298465 19517 heartbeater.cc:464] Registering TS with master...
I20260925 09:17:41.299181 19517 heartbeater.cc:511] Master 127.3.19.126:46573 requested a full tablet report, sending...
I20260925 09:17:41.301388 19332 ts_manager.cc:194] Registered new tserver with Master: 16494a03f1a046f58757e4f7b029e0e3 (127.3.19.65:37555)
I20260925 09:17:41.302933 19332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:45465
W20260925 09:17:41.603369 19521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:17:41.603675 19521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:17:41.603840 19521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:17:41.610896 19521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:17:41.611043 19521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:17:41.611218 19521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:17:41.620697 19521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:37315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=35665
--tserver_master_addrs=127.3.19.126:46573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:17:41.621943 19521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:17:41.623179 19521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:17:41.635475 19526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:41.635459 19527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:41.635437 19529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:17:41.636389 19521 server_base.cc:1034] running on GCE node
I20260925 09:17:41.636904 19521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:17:41.637799 19521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:17:41.638989 19521 hybrid_clock.cc:648] HybridClock initialized: now 1790327861638955 us; error 31 us; skew 500 ppm
I20260925 09:17:41.639298 19521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:17:41 dist-test-slave-l6qh krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327861, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:17:41.642328 19521 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:17:41.643764 19521 webserver.cc:460] Webserver started at http://127.3.19.66:35665/ using document root <none> and password file <none>
I20260925 09:17:41.644398 19521 fs_manager.cc:359] Metadata directory not provided
I20260925 09:17:41.644557 19521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:17:41.649969 19521 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 09:17:41.653781 19536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:41.655475 19521 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260925 09:17:41.655640 19521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "2196952eacd849eb8a0670f02dae647a"
format_stamp: "Formatted at 2026-09-25 09:17:18 on dist-test-slave-l6qh"
server_key: "59a5b4ad0359eaac4077c7de1d0ec529"
server_key_iv: "4ab09387d2a0968a21b22b239e931260"
server_key_version: "encryptionkey@0"
I20260925 09:17:41.656956 19521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:17:41.681164 19521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:17:41.685302 19521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:17:41.685554 19521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:17:41.686475 19521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:17:41.687960 19521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:17:41.688090 19521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:41.688211 19521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:17:41.688323 19521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:41.721601 19521 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:37315
I20260925 09:17:41.721691 19648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:37315 every 8 connection(s)
I20260925 09:17:41.722980 19521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:17:41.724341 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 19521
I20260925 09:17:41.724920 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:37195
--local_ip_for_outbound_sockets=127.3.19.67
--tserver_master_addrs=127.3.19.126:46573
--webserver_port=46051
--webserver_interface=127.3.19.67
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 09:17:41 dist-test-slave-l6qh krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327861, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:17:41.744786 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:41.725394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:58673 (local address 127.3.19.126:46573)
0925 09:17:41.726938 (+ 1544us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:41.726942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:41.726960 (+ 18us) server_negotiation.cc:407] Connection header received
0925 09:17:41.729484 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:41.729488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:41.729550 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:41.729630 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:41.731647 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:41.732335 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:41.734508 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:41.734966 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:41.735104 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:41.739009 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:41.739014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:41.739017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:41.739047 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:41.741616 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:41.742463 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:41.742468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:41.742470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:41.742601 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:41.743028 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:41.743032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:41.743034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:41.743427 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:41.743576 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:41.744233 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 09:17:41.744451 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1437}
I20260925 09:17:41.745803 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:41.725850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:58673)
0925 09:17:41.726706 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:41.726758 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:41.729182 (+ 2424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:41.729859 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:41.729872 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:41.730670 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:17:41.731467 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:41.731494 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:41.732483 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:41.732491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:41.734366 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:41.734378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:41.735194 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:41.735202 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:41.735523 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:41.736251 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:17:41.736283 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:17:41.738791 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:17:41.741784 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:41.741791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:41.741806 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:41.742313 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:41.742718 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:41.742724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:41.742728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:41.742906 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:41.743547 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:17:41.743555 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:17:41.743885 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 09:17:41.744144 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 09:17:41.744640 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":308,"threads_started":1}
I20260925 09:17:41.747143 19649 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46573
I20260925 09:17:41.747450 19649 heartbeater.cc:464] Registering TS with master...
I20260925 09:17:41.748106 19649 heartbeater.cc:511] Master 127.3.19.126:46573 requested a full tablet report, sending...
I20260925 09:17:41.749533 19332 ts_manager.cc:194] Registered new tserver with Master: 2196952eacd849eb8a0670f02dae647a (127.3.19.66:37315)
I20260925 09:17:41.750301 19332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:58673
W20260925 09:17:42.052042 19652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:17:42.052335 19652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:17:42.052500 19652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:17:42.059492 19652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:17:42.059649 19652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:17:42.059767 19652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:17:42.068895 19652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:37195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=46051
--tserver_master_addrs=127.3.19.126:46573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:17:42.070091 19652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:17:42.071347 19652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:17:42.083719 19657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:42.083818 19660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:17:42.083890 19658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:17:42.085039 19652 server_base.cc:1034] running on GCE node
I20260925 09:17:42.085704 19652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:17:42.086591 19652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:17:42.087813 19652 hybrid_clock.cc:648] HybridClock initialized: now 1790327862087739 us; error 74 us; skew 500 ppm
I20260925 09:17:42.088119 19652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:17:42 dist-test-slave-l6qh krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327862, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:17:42.091235 19652 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:17:42.092693 19652 webserver.cc:460] Webserver started at http://127.3.19.67:46051/ using document root <none> and password file <none>
I20260925 09:17:42.093446 19652 fs_manager.cc:359] Metadata directory not provided
I20260925 09:17:42.093606 19652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:17:42.098555 19652 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 09:17:42.101140 19667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:42.102524 19652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 09:17:42.102665 19652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "696fe6f9a0eb4ab7b9845853f1903b3e"
format_stamp: "Formatted at 2026-09-25 09:17:19 on dist-test-slave-l6qh"
server_key: "dc77e433ae6a48cd2075028676990ae5"
server_key_iv: "62deb258ab24e48f4e4e8c62b507f6f9"
server_key_version: "encryptionkey@0"
I20260925 09:17:42.103689 19652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:17:42.133013 19652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:17:42.136828 19652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:17:42.137039 19652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:17:42.137909 19652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:17:42.139146 19652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:17:42.139227 19652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:42.139299 19652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:17:42.139353 19652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:17:42.172842 19652 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:37195
I20260925 09:17:42.172926 19779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:37195 every 8 connection(s)
I20260925 09:17:42.174186 19652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:17:42.181615 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 19652
Sep 25 09:17:42 dist-test-slave-l6qh krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327862, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:17:42.195935 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.176553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:50631 (local address 127.3.19.126:46573)
0925 09:17:42.176724 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:42.176728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:42.177817 (+ 1089us) server_negotiation.cc:407] Connection header received
0925 09:17:42.179666 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:42.179680 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:42.179770 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:42.179881 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:42.181321 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.182133 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.184147 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.184893 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.185043 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.188787 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:42.188795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:42.188798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:42.188828 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:42.192564 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:42.193529 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:42.193535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:42.193540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:42.193686 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:42.194143 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:42.194147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:42.194149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:42.194487 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:42.194672 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:42.195483 (+ 811us) server_negotiation.cc:299] Negotiation successful
0925 09:17:42.195682 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 09:17:42.197268 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.176908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:50631)
0925 09:17:42.177654 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:42.177696 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:42.179388 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:42.180109 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:42.180116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:42.180601 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:17:42.181125 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:42.181141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.182322 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.182327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:42.183980 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:42.184000 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.185090 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.185095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:42.185340 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.185955 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:17:42.185980 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:17:42.188599 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:17:42.192696 (+ 4097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:42.192704 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:42.192719 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:42.193386 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:42.193796 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:17:42.193803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:17:42.193808 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:17:42.194034 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:17:42.194642 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:17:42.194651 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:17:42.195057 (+ 406us) client_negotiation.cc:769] Sending connection context
0925 09:17:42.195395 (+ 338us) client_negotiation.cc:240] Negotiation successful
0925 09:17:42.195921 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":359,"threads_started":1}
I20260925 09:17:42.198827 19780 heartbeater.cc:347] Connected to a master server at 127.3.19.126:46573
I20260925 09:17:42.199113 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.182496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.3.19.126:46573)
0925 09:17:42.182985 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:42.182990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:42.183332 (+ 342us) server_negotiation.cc:407] Connection header received
0925 09:17:42.183459 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:42.183464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:42.183538 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:42.183642 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:42.185668 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.186706 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.188309 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.188924 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.189010 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.192614 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:42.192641 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:42.192651 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:42.192690 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:42.196216 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:42.196897 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:42.196902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:42.196905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:42.197034 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:42.197592 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:42.197597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:42.197600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:42.197955 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:42.198200 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:42.198545 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 09:17:42.198711 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":240,"threads_started":1}
I20260925 09:17:42.199141 19780 heartbeater.cc:464] Registering TS with master...
I20260925 09:17:42.199836 19780 heartbeater.cc:511] Master 127.3.19.126:46573 requested a full tablet report, sending...
I20260925 09:17:42.201545 19331 ts_manager.cc:194] Registered new tserver with Master: 696fe6f9a0eb4ab7b9845853f1903b3e (127.3.19.67:37195)
I20260925 09:17:42.202489 19331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:50631
I20260925 09:17:42.213886 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:17:42.220247 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.214704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39230 (local address 127.3.19.126:46573)
0925 09:17:42.214902 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:42.214909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:42.215484 (+ 575us) server_negotiation.cc:407] Connection header received
0925 09:17:42.215649 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:42.215653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:42.215806 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:42.215948 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:17:42.216373 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.217053 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.218570 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.218992 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.219081 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.219558 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:17:42.219780 (+ 222us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 09:17:42.220036 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39230: token expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 09:17:42.220376 19783 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39230: token expired
W20260925 09:17:42.220932 19254 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.19.126:46573)
I20260925 09:17:42.233117 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.221476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.3.19.126:46573)
0925 09:17:42.221615 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:42.221619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:42.221670 (+ 51us) server_negotiation.cc:407] Connection header received
0925 09:17:42.221808 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:42.221811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:42.221860 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:42.221933 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:42.223189 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.223870 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.225079 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.225540 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.225630 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.228078 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:42.228082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:42.228085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:42.228110 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:42.230524 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:42.231178 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:42.231185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:42.231194 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:42.231324 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:42.231712 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:42.231717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:42.231721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:42.232007 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:42.232278 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:42.232572 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 09:17:42.232798 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 09:17:42.236333 19331 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:17:42.243706 19331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:17:42.278631 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.258318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55474 (local address 127.3.19.67:37195)
0925 09:17:42.270210 (+ 11892us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:42.270230 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:42.270263 (+ 33us) server_negotiation.cc:407] Connection header received
0925 09:17:42.270370 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:42.270377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:42.270602 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:42.271567 (+ 965us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:17:42.271937 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.273556 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.276394 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.277444 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.277583 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.277847 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:42.278141 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 09:17:42.278299 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11732,"thread_start_us":11595,"threads_started":1}
I20260925 09:17:42.278829 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.257529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:37195 (local address 127.0.0.1:55474)
0925 09:17:42.260620 (+ 3091us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:42.260674 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:42.260886 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:42.270869 (+ 9983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:42.270878 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:42.270908 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:17:42.271340 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:42.271354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.273707 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.273712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:42.276211 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:42.276237 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.277654 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.277660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:42.277883 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.277938 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 09:17:42.278095 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 09:17:42.278285 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2869,"mutex_wait_us":51,"thread_start_us":2748,"threads_started":1}
I20260925 09:17:42.279100 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.257758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:37315 (local address 127.0.0.1:48966)
0925 09:17:42.265732 (+ 7974us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:42.265758 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:42.265881 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:42.271548 (+ 5667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:42.271552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:42.271577 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:17:42.272019 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:42.272034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.274062 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.274069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:42.275817 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:42.275834 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.277998 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.278006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:42.278200 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.278231 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 09:17:42.278318 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 09:17:42.278464 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7826,"mutex_wait_us":32,"thread_start_us":7666,"threads_started":1}
I20260925 09:17:42.279817 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.258749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.3.19.66:37315)
0925 09:17:42.271115 (+ 12366us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:42.271121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:42.271142 (+ 21us) server_negotiation.cc:407] Connection header received
0925 09:17:42.271216 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:42.271221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:42.271373 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:42.271565 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:17:42.272378 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.273943 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.276011 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.277471 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.277589 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.277830 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:42.279331 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0925 09:17:42.279511 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12220,"thread_start_us":12087,"threads_started":1}
I20260925 09:17:42.281775 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.258220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:37555 (local address 127.0.0.1:39716)
0925 09:17:42.272282 (+ 14062us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:42.272300 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:42.272403 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:42.273305 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:42.273309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:42.273331 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:17:42.273650 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:42.273660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.275530 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.275535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:42.278606 (+ 3071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:42.278620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.281076 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.281083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:42.281262 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.281292 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 09:17:42.281367 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 09:17:42.281477 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13869,"thread_start_us":13708,"threads_started":1}
I20260925 09:17:42.282577 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.258859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.3.19.65:37555)
0925 09:17:42.269761 (+ 10902us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:42.269770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:42.272523 (+ 2753us) server_negotiation.cc:407] Connection header received
0925 09:17:42.272615 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:42.272621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:42.272825 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:42.273026 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:17:42.273811 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.275353 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.278812 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.280585 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.280719 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.280933 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:42.282107 (+ 1174us) server_negotiation.cc:299] Negotiation successful
0925 09:17:42.282263 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10734,"thread_start_us":12387,"threads_started":1}
I20260925 09:17:42.282933 19715 tablet_service.cc:1467] Processing CreateTablet for tablet 3bd887b479f74544b7e436d3826e62ac (DEFAULT_TABLE table=test-table [id=06bc1dc8dbb445d39bdf9e23a11cf5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:17:42.284319 19715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd887b479f74544b7e436d3826e62ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:17:42.284106 19584 tablet_service.cc:1467] Processing CreateTablet for tablet 3bd887b479f74544b7e436d3826e62ac (DEFAULT_TABLE table=test-table [id=06bc1dc8dbb445d39bdf9e23a11cf5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:17:42.285473 19584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd887b479f74544b7e436d3826e62ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:17:42.285897 19452 tablet_service.cc:1467] Processing CreateTablet for tablet 3bd887b479f74544b7e436d3826e62ac (DEFAULT_TABLE table=test-table [id=06bc1dc8dbb445d39bdf9e23a11cf5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:17:42.287261 19452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd887b479f74544b7e436d3826e62ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:17:42.297173 19796 tablet_bootstrap.cc:492] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e: Bootstrap starting.
I20260925 09:17:42.300261 19797 tablet_bootstrap.cc:492] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a: Bootstrap starting.
I20260925 09:17:42.300498 19796 tablet_bootstrap.cc:654] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e: Neither blocks nor log segments found. Creating new log.
I20260925 09:17:42.301687 19798 tablet_bootstrap.cc:492] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: Bootstrap starting.
I20260925 09:17:42.302004 19796 log.cc:824] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e: Log is configured to *not* fsync() on all Append() calls
I20260925 09:17:42.303843 19797 tablet_bootstrap.cc:654] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a: Neither blocks nor log segments found. Creating new log.
I20260925 09:17:42.304628 19798 tablet_bootstrap.cc:654] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: Neither blocks nor log segments found. Creating new log.
I20260925 09:17:42.305212 19797 log.cc:824] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a: Log is configured to *not* fsync() on all Append() calls
I20260925 09:17:42.305637 19798 log.cc:824] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: Log is configured to *not* fsync() on all Append() calls
I20260925 09:17:42.305951 19517 heartbeater.cc:503] Master 127.3.19.126:46573 was elected leader, sending a full tablet report...
I20260925 09:17:42.308120 19797 tablet_bootstrap.cc:492] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a: No bootstrap required, opened a new log
I20260925 09:17:42.308120 19796 tablet_bootstrap.cc:492] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e: No bootstrap required, opened a new log
I20260925 09:17:42.308358 19797 ts_tablet_manager.cc:1397] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260925 09:17:42.308358 19796 ts_tablet_manager.cc:1397] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260925 09:17:42.309270 19798 tablet_bootstrap.cc:492] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: No bootstrap required, opened a new log
I20260925 09:17:42.309494 19798 ts_tablet_manager.cc:1397] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 09:17:42.316761 19798 raft_consensus.cc:348] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:42.317086 19798 raft_consensus.cc:374] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:17:42.317176 19798 raft_consensus.cc:729] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16494a03f1a046f58757e4f7b029e0e3, State: Initialized, Role: FOLLOWER
I20260925 09:17:42.317768 19798 consensus_queue.cc:260] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:42.318727 19797 raft_consensus.cc:348] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:42.319098 19797 raft_consensus.cc:374] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:17:42.319211 19797 raft_consensus.cc:729] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2196952eacd849eb8a0670f02dae647a, State: Initialized, Role: FOLLOWER
I20260925 09:17:42.319424 19798 ts_tablet_manager.cc:1428] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260925 09:17:42.319438 19796 raft_consensus.cc:348] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:42.319916 19796 raft_consensus.cc:374] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:17:42.320019 19796 raft_consensus.cc:729] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696fe6f9a0eb4ab7b9845853f1903b3e, State: Initialized, Role: FOLLOWER
I20260925 09:17:42.319999 19797 consensus_queue.cc:260] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:42.320588 19796 consensus_queue.cc:260] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:42.321141 19649 heartbeater.cc:503] Master 127.3.19.126:46573 was elected leader, sending a full tablet report...
I20260925 09:17:42.321900 19797 ts_tablet_manager.cc:1428] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260925 09:17:42.322753 19780 heartbeater.cc:503] Master 127.3.19.126:46573 was elected leader, sending a full tablet report...
I20260925 09:17:42.323328 19796 ts_tablet_manager.cc:1428] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260925 09:17:42.327937 19802 raft_consensus.cc:482] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:17:42.328154 19802 raft_consensus.cc:504] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:42.329661 19802 leader_election.cc:288] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696fe6f9a0eb4ab7b9845853f1903b3e (127.3.19.67:37195), 2196952eacd849eb8a0670f02dae647a (127.3.19.66:37315)
I20260925 09:17:42.336277 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.329910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:47125 (local address 127.3.19.67:37195)
0925 09:17:42.330076 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:42.330084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:42.330900 (+ 816us) server_negotiation.cc:407] Connection header received
0925 09:17:42.331098 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:42.331104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:42.331169 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:42.331276 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:17:42.331829 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.332549 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.334396 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.335426 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.335561 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.335660 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:42.335925 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 09:17:42.336036 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 09:17:42.336431 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.330020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:52011 (local address 127.3.19.66:37315)
0925 09:17:42.330201 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:42.330207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:42.331014 (+ 807us) server_negotiation.cc:407] Connection header received
0925 09:17:42.331197 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:42.331203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:42.331269 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:42.331363 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:17:42.331962 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.332656 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.334606 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.335588 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.335741 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.335832 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:42.336091 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 09:17:42.336203 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 09:17:42.336553 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.330062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:37315 (local address 127.3.19.65:52011)
0925 09:17:42.330915 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:42.330929 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:42.331075 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:42.331392 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:42.331397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:42.331436 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:17:42.331806 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:42.331813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.332809 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.332822 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:42.334445 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:42.334459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.335798 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.335804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:42.335939 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.335961 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 09:17:42.336057 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 09:17:42.336177 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":2,"thread_start_us":446,"threads_started":1}
I20260925 09:17:42.336807 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.330031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:37195 (local address 127.3.19.65:47125)
0925 09:17:42.330802 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:42.330817 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:17:42.330972 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:17:42.331294 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:17:42.331299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:17:42.331323 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:17:42.331686 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:42.331698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.332684 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.332692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:42.334246 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:17:42.334258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.335611 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.335618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:17:42.335778 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.335801 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 09:17:42.335900 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 09:17:42.336024 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":31,"thread_start_us":403,"threads_started":1}
I20260925 09:17:42.337499 19604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd887b479f74544b7e436d3826e62ac" candidate_uuid: "16494a03f1a046f58757e4f7b029e0e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2196952eacd849eb8a0670f02dae647a" is_pre_election: true
I20260925 09:17:42.337661 19735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd887b479f74544b7e436d3826e62ac" candidate_uuid: "16494a03f1a046f58757e4f7b029e0e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" is_pre_election: true
I20260925 09:17:42.337953 19604 raft_consensus.cc:2454] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16494a03f1a046f58757e4f7b029e0e3 in term 0.
I20260925 09:17:42.338135 19735 raft_consensus.cc:2454] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16494a03f1a046f58757e4f7b029e0e3 in term 0.
I20260925 09:17:42.338636 19407 leader_election.cc:302] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16494a03f1a046f58757e4f7b029e0e3, 2196952eacd849eb8a0670f02dae647a; no voters:
I20260925 09:17:42.339011 19802 raft_consensus.cc:2781] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:17:42.339155 19802 raft_consensus.cc:482] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:17:42.339267 19802 raft_consensus.cc:3037] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:17:42.341471 19802 raft_consensus.cc:504] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:42.342090 19802 leader_election.cc:288] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [CANDIDATE]: Term 1 election: Requested vote from peers 696fe6f9a0eb4ab7b9845853f1903b3e (127.3.19.67:37195), 2196952eacd849eb8a0670f02dae647a (127.3.19.66:37315)
I20260925 09:17:42.342476 19735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd887b479f74544b7e436d3826e62ac" candidate_uuid: "16494a03f1a046f58757e4f7b029e0e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e"
I20260925 09:17:42.342504 19604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd887b479f74544b7e436d3826e62ac" candidate_uuid: "16494a03f1a046f58757e4f7b029e0e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2196952eacd849eb8a0670f02dae647a"
I20260925 09:17:42.342660 19735 raft_consensus.cc:3037] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:17:42.342698 19604 raft_consensus.cc:3037] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:17:42.344900 19735 raft_consensus.cc:2454] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16494a03f1a046f58757e4f7b029e0e3 in term 1.
I20260925 09:17:42.344910 19604 raft_consensus.cc:2454] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16494a03f1a046f58757e4f7b029e0e3 in term 1.
I20260925 09:17:42.345335 19405 leader_election.cc:302] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16494a03f1a046f58757e4f7b029e0e3, 696fe6f9a0eb4ab7b9845853f1903b3e; no voters:
I20260925 09:17:42.345618 19802 raft_consensus.cc:2781] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:17:42.346005 19802 raft_consensus.cc:686] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [term 1 LEADER]: Becoming Leader. State: Replica: 16494a03f1a046f58757e4f7b029e0e3, State: Running, Role: LEADER
I20260925 09:17:42.346439 19802 consensus_queue.cc:237] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } } peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } }
I20260925 09:17:42.350739 19332 catalog_manager.cc:5495] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16494a03f1a046f58757e4f7b029e0e3 (127.3.19.65). New cstate: current_term: 1 leader_uuid: "16494a03f1a046f58757e4f7b029e0e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16494a03f1a046f58757e4f7b029e0e3" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 37555 } health_report { overall_health: HEALTHY } } }
I20260925 09:17:42.381399 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.376441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.3.19.65:37555)
0925 09:17:42.376627 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:42.376631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:42.376648 (+ 17us) server_negotiation.cc:407] Connection header received
0925 09:17:42.376848 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:42.376852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:42.376927 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:42.377051 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:17:42.377354 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.378081 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.379538 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.379918 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.380004 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.380424 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:17:42.380885 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:17:42.380987 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:42.381080 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 09:17:42.381139 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 09:17:42.389112 19735 raft_consensus.cc:1261] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e [term 1 FOLLOWER]: Refusing update from remote peer 16494a03f1a046f58757e4f7b029e0e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:17:42.389111 19604 raft_consensus.cc:1261] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a [term 1 FOLLOWER]: Refusing update from remote peer 16494a03f1a046f58757e4f7b029e0e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:17:42.390093 19802 consensus_queue.cc:1035] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2196952eacd849eb8a0670f02dae647a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:17:42.390368 19807 consensus_queue.cc:1035] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696fe6f9a0eb4ab7b9845853f1903b3e" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:17:42.404043 19811 mvcc.cc:205] Tried to move back new op lower bound from 7333182924336500736 to 7333182924174508032. Current Snapshot: MvccSnapshot[applied={T|T < 7333182924336500736}]
I20260925 09:17:42.416227 19331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:17:42.416491 19331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:17:42.419540 19331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Sending DeleteTablet for 3 replicas of tablet 3bd887b479f74544b7e436d3826e62ac
I20260925 09:17:42.420495 19715 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bd887b479f74544b7e436d3826e62ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:17:42 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:55474
I20260925 09:17:42.420619 19584 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bd887b479f74544b7e436d3826e62ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:17:42 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:48966
I20260925 09:17:42.420917 19452 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bd887b479f74544b7e436d3826e62ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:17:42 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:39716
I20260925 09:17:42.422034 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 19369
I20260925 09:17:42.431562 19820 tablet_replica.cc:330] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e: stopping tablet replica
I20260925 09:17:42.432067 19820 raft_consensus.cc:2229] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:17:42.432629 19820 raft_consensus.cc:2258] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:17:42.434067 19821 tablet_replica.cc:330] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a: stopping tablet replica
I20260925 09:17:42.434444 19821 raft_consensus.cc:2229] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:17:42.434886 19821 raft_consensus.cc:2258] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:17:42.439653 19820 ts_tablet_manager.cc:1905] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:17:42.440546 19821 ts_tablet_manager.cc:1905] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:17:42.444303 19822 tablet_replica.cc:330] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: stopping tablet replica
I20260925 09:17:42.444706 19822 raft_consensus.cc:2229] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:17:42.445359 19822 raft_consensus.cc:2258] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:17:42.447208 19822 ts_tablet_manager.cc:1905] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:17:42.447790 19820 ts_tablet_manager.cc:1918] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:17:42.447942 19820 log.cc:1196] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/3bd887b479f74544b7e436d3826e62ac
I20260925 09:17:42.448486 19820 ts_tablet_manager.cc:1939] T 3bd887b479f74544b7e436d3826e62ac P 696fe6f9a0eb4ab7b9845853f1903b3e: Deleting consensus metadata
I20260925 09:17:42.450325 19316 catalog_manager.cc:4841] TS 696fe6f9a0eb4ab7b9845853f1903b3e (127.3.19.67:37195): tablet 3bd887b479f74544b7e436d3826e62ac (table test-table [id=06bc1dc8dbb445d39bdf9e23a11cf5bc]) successfully deleted
I20260925 09:17:42.450956 19821 ts_tablet_manager.cc:1918] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:17:42.451107 19821 log.cc:1196] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/3bd887b479f74544b7e436d3826e62ac
I20260925 09:17:42.451629 19821 ts_tablet_manager.cc:1939] T 3bd887b479f74544b7e436d3826e62ac P 2196952eacd849eb8a0670f02dae647a: Deleting consensus metadata
I20260925 09:17:42.453011 19318 catalog_manager.cc:4841] TS 2196952eacd849eb8a0670f02dae647a (127.3.19.66:37315): tablet 3bd887b479f74544b7e436d3826e62ac (table test-table [id=06bc1dc8dbb445d39bdf9e23a11cf5bc]) successfully deleted
I20260925 09:17:42.455039 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.422942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.3.19.65:37555)
0925 09:17:42.423149 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:42.423158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:42.423262 (+ 104us) server_negotiation.cc:407] Connection header received
0925 09:17:42.423468 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:42.423476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:42.423572 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:42.423690 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:42.425947 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.427330 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.429792 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.430726 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.430883 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.435807 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:42.435823 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:42.435838 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:42.435879 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:42.440509 (+ 4630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:42.441481 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:42.441491 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:42.441497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:42.441667 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:42.453326 (+ 11659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:42.453333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:42.453337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:42.453794 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:42.454104 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:42.454389 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 09:17:42.454659 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 09:17:42.455799 19512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:17:42.550863 19822 ts_tablet_manager.cc:1918] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:17:42.551190 19822 log.cc:1196] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/3bd887b479f74544b7e436d3826e62ac
I20260925 09:17:42.551575 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 19369
W20260925 09:17:42.571364 19317 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.19.65:37555 (error 108)
I20260925 09:17:42.571587 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 19521
I20260925 09:17:42.573391 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.572913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:37555 (local address 127.0.0.1:39744)
0925 09:17:42.573079 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:42.573189 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 09:17:42.573644 19317 catalog_manager.cc:4584] TS 16494a03f1a046f58757e4f7b029e0e3 (127.3.19.65:37555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3bd887b479f74544b7e436d3826e62ac: Network error: Client connection negotiation failed: client connection to 127.3.19.65:37555: connect: Connection refused (error 111)
I20260925 09:17:42.585000 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.572379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48970 (local address 127.3.19.66:37315)
0925 09:17:42.572601 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:42.572607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:42.572749 (+ 142us) server_negotiation.cc:407] Connection header received
0925 09:17:42.572919 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:42.572923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:42.572994 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:42.573119 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:42.574636 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.575369 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.576670 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.577067 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.577201 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.579688 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:42.579699 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:42.579713 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:42.579757 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:42.582355 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:42.583037 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:42.583044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:42.583046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:42.583163 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:42.583528 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:42.583533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:42.583536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:42.583888 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:42.584175 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:42.584475 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 09:17:42.584685 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 09:17:42.585750 19644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:17:42.627475 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.626909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:37555 (local address 127.0.0.1:39748)
0925 09:17:42.627172 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:42.627280 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 09:17:42.628496 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.628135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:37555 (local address 127.0.0.1:39752)
0925 09:17:42.628285 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:42.628362 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 09:17:42.672138 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 19521
I20260925 09:17:42.688987 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.688501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:37555 (local address 127.0.0.1:39760)
0925 09:17:42.688741 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:42.688824 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 09:17:42.690092 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 19652
I20260925 09:17:42.690173 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.689823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:37555 (local address 127.0.0.1:39774)
0925 09:17:42.689972 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:42.690044 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 09:17:42.703378 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.690842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.3.19.67:37195)
0925 09:17:42.691022 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 09:17:42.691030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:17:42.691054 (+ 24us) server_negotiation.cc:407] Connection header received
0925 09:17:42.691262 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:17:42.691268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:17:42.691342 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:17:42.691466 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:17:42.692827 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.693645 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.694914 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:17:42.695280 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:17:42.695422 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:17:42.698050 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:17:42.698068 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:17:42.698087 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:17:42.698125 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:17:42.700633 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:42.701376 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:42.701389 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:42.701391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:42.701520 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:17:42.701902 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:17:42.701908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:17:42.701911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:17:42.702251 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:17:42.702538 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 09:17:42.702844 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 09:17:42.703059 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 09:17:42.703997 19775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:17:42.770795 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.770323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:37555 (local address 127.0.0.1:39776)
0925 09:17:42.770531 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:42.770615 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 09:17:42.771930 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:17:42.771516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:37555 (local address 127.0.0.1:39782)
0925 09:17:42.771728 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 09:17:42.771809 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 09:17:42.791286 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 19652
I20260925 09:17:42.809742 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 19299
I20260925 09:17:42.810847 19362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:17:42.922847 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 19299
I20260925 09:18:02.945544 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46573
--webserver_interface=127.3.19.126
--webserver_port=37393
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:46573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:03.278088 19828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:03.278414 19828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:03.278568 19828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:03.286293 19828 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 09:18:03.286413 19828 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 09:18:03.286480 19828 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 09:18:03.286552 19828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:03.286657 19828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:03.286760 19828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:03.286854 19828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:03.286931 19828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:03.296693 19828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:46573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:46573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=37393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:03.298064 19828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:03.299404 19828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:03.311693 19834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:03.311657 19836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:03.311736 19833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:03.312106 19828 server_base.cc:1034] running on GCE node
I20260925 09:18:03.312992 19828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:03.314389 19828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:03.315654 19828 hybrid_clock.cc:648] HybridClock initialized: now 1790327883315586 us; error 37 us; skew 500 ppm
I20260925 09:18:03.316025 19828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:03 dist-test-slave-l6qh krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327883, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:03.319056 19828 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:18:03.320447 19828 webserver.cc:460] Webserver started at http://127.3.19.126:37393/ using document root <none> and password file <none>
I20260925 09:18:03.321132 19828 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:03.321281 19828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:03.327087 19828 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 09:18:03.329802 19843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:03.331045 19828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:18:03.331223 19828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "1c76d36dcf4749e5888ae8ab5e896e43"
format_stamp: "Formatted at 2026-09-25 09:17:17 on dist-test-slave-l6qh"
server_key: "270bc535692164dbb5798141f74570bc"
server_key_iv: "dd1dff53dc4e04b39962fd85db131d6d"
server_key_version: "encryptionkey@0"
I20260925 09:18:03.332149 19828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:03.350689 19828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:03.354480 19828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:03.354749 19828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:03.373029 19828 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:46573
I20260925 09:18:03.373075 19894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:46573 every 8 connection(s)
I20260925 09:18:03.374470 19828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:03.379034 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 19828
I20260925 09:18:03.380054 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:37555
--local_ip_for_outbound_sockets=127.3.19.65
--tserver_master_addrs=127.3.19.126:46573
--webserver_port=45013
--webserver_interface=127.3.19.65
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:18:03.381953 19895 sys_catalog.cc:263] Verifying existing consensus state
I20260925 09:18:03.387044 19895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Bootstrap starting.
I20260925 09:18:03.408241 19895 log.cc:824] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:03.423774 19895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 09:18:03.424258 19895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Bootstrap complete.
I20260925 09:18:03.432966 19895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } }
I20260925 09:18:03.433854 19895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c76d36dcf4749e5888ae8ab5e896e43, State: Initialized, Role: FOLLOWER
I20260925 09:18:03.434403 19895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } }
I20260925 09:18:03.434617 19895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:03.434739 19895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:03.434931 19895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 2 FOLLOWER]: Advancing to term 3
I20260925 09:18:03.438911 19895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } }
I20260925 09:18:03.439411 19895 leader_election.cc:302] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c76d36dcf4749e5888ae8ab5e896e43; no voters:
I20260925 09:18:03.440112 19895 leader_election.cc:288] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 09:18:03.440246 19899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 09:18:03.441537 19899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [term 3 LEADER]: Becoming Leader. State: Replica: 1c76d36dcf4749e5888ae8ab5e896e43, State: Running, Role: LEADER
I20260925 09:18:03.442169 19899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } }
I20260925 09:18:03.442390 19895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:03.444769 19900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } } }
I20260925 09:18:03.445188 19900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:03.444742 19901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c76d36dcf4749e5888ae8ab5e896e43. Latest consensus state: current_term: 3 leader_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c76d36dcf4749e5888ae8ab5e896e43" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 46573 } } }
I20260925 09:18:03.446043 19901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:03.453811 19905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:03.461958 19905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:03.463410 19905 catalog_manager.cc:1259] Loaded cluster ID: 273845f7479c4c5395cd989713e3bbfb
I20260925 09:18:03.463567 19905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:03.468030 19905 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:03.469537 19905 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Loaded TSK: 0
I20260925 09:18:03.469744 19905 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c76d36dcf4749e5888ae8ab5e896e43: Loaded TSK: 1
F20260925 09:18:03.470782 19905 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7faa6d9699ad google::LogMessage::Fail() at ??:0
@ 0x7faa6d96c88a google::LogMessage::SendToLog() at ??:0
@ 0x7faa6d9694de google::LogMessage::Flush() at ??:0
@ 0x7faa6d96aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7faa716cd7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7faa716bc73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7faa8339c1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7faa8339b783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7faa834272a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7faa834270c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7faa834715e9 std::function<>::operator()() at ??:0
@ 0x7faa8339fcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
I20260925 09:18:03.522085 19919 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7faa8339dc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7faa83425610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7faa83425439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7faa705b5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faa705b891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7faa705b8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7faa70593eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7faa7386d6db start_thread at ??:0
@ 0x7faa6c80361f clone at ??:0
W20260925 09:18:03.789124 19897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:03.789390 19897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:03.789553 19897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:03.797338 19897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:03.797494 19897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:03.797739 19897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:18:03.808492 19897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:37555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=45013
--tserver_master_addrs=127.3.19.126:46573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:03.809855 19897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:03.811300 19897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:03.824981 19942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:03.825021 19943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:03.825057 19945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:03.825821 19897 server_base.cc:1034] running on GCE node
I20260925 09:18:03.826514 19897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:03.827512 19897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:03.828823 19897 hybrid_clock.cc:648] HybridClock initialized: now 1790327883828760 us; error 44 us; skew 500 ppm
I20260925 09:18:03.829178 19897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:03 dist-test-slave-l6qh krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327883, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:03.832530 19897 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:18:03.834048 19897 webserver.cc:460] Webserver started at http://127.3.19.65:45013/ using document root <none> and password file <none>
I20260925 09:18:03.834714 19897 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:03.834859 19897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:03.840085 19897 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 09:18:03.842724 19952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:03.843889 19897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 09:18:03.844058 19897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "16494a03f1a046f58757e4f7b029e0e3"
format_stamp: "Formatted at 2026-09-25 09:17:18 on dist-test-slave-l6qh"
server_key: "18cc30db1971e854254ca0b0671a405b"
server_key_iv: "f7e825d37fb328736bdb5955df671c93"
server_key_version: "encryptionkey@0"
I20260925 09:18:03.845010 19897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:03.884343 19897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:03.888268 19897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:03.888545 19897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:03.889544 19897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:03.891217 19959 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260925 09:18:03.896400 19959 ts_tablet_manager.cc:1852] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 09:18:03.897497 19959 ts_tablet_manager.cc:1905] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:03.904632 19959 ts_tablet_manager.cc:1918] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:03.904878 19959 ts_tablet_manager.cc:1939] T 3bd887b479f74544b7e436d3826e62ac P 16494a03f1a046f58757e4f7b029e0e3: Deleting consensus metadata
I20260925 09:18:03.905282 19897 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 09:18:03.905421 19897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.000s
I20260925 09:18:03.905584 19897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:03.905697 19897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:03.940274 19897 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:37555
I20260925 09:18:03.940361 20065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:37555 every 8 connection(s)
I20260925 09:18:03.941675 19897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:18:03.945690 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 19897
I20260925 09:18:03.946498 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:37315
--local_ip_for_outbound_sockets=127.3.19.66
--tserver_master_addrs=127.3.19.126:46573
--webserver_port=35665
--webserver_interface=127.3.19.66
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:18:03.947227 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:03.944689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:55437)
0925 09:18:03.945724 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:03.946007 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":368,"threads_started":1}
W20260925 09:18:03.947975 20066 heartbeater.cc:644] Failed to heartbeat to 127.3.19.126:46573 (0 consecutive failures): Network error: Failed to ping master at 127.3.19.126:46573: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
I20260925 09:18:03.948897 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:03.948514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:34627)
0925 09:18:03.948659 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:03.948742 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 09:18:03.949934 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:03.949559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:35993)
0925 09:18:03.949726 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:03.949817 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 09:18:03.950224 20066 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 09:18:04.276727 20069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:04.277007 20069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:04.277102 20069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:04.284459 20069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:04.284565 20069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:04.284680 20069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:18:04.294826 20069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:37315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=35665
--tserver_master_addrs=127.3.19.126:46573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:04.296147 20069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:04.297518 20069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:04.311074 20074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:04.311215 20075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:04.311249 20077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:04.312500 20069 server_base.cc:1034] running on GCE node
I20260925 09:18:04.313176 20069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:04.314210 20069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:04.315418 20069 hybrid_clock.cc:648] HybridClock initialized: now 1790327884315363 us; error 48 us; skew 500 ppm
I20260925 09:18:04.315717 20069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:04 dist-test-slave-l6qh krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327884, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:04.318990 20069 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:18:04.320600 20069 webserver.cc:460] Webserver started at http://127.3.19.66:35665/ using document root <none> and password file <none>
I20260925 09:18:04.321384 20069 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:04.321517 20069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:04.327087 20069 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 09:18:04.330003 20084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:04.331523 20069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:18:04.331713 20069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "2196952eacd849eb8a0670f02dae647a"
format_stamp: "Formatted at 2026-09-25 09:17:18 on dist-test-slave-l6qh"
server_key: "59a5b4ad0359eaac4077c7de1d0ec529"
server_key_iv: "4ab09387d2a0968a21b22b239e931260"
server_key_version: "encryptionkey@0"
I20260925 09:18:04.332734 20069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:04.350446 20069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:04.355122 20069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:04.355432 20069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:04.356482 20069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:04.358019 20069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:04.358147 20069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:04.358287 20069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:04.358376 20069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:04.394068 20069 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:37315
I20260925 09:18:04.394114 20196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:37315 every 8 connection(s)
I20260925 09:18:04.395584 20069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:18:04.400794 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:04.398796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:48099)
0925 09:18:04.399645 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:04.399875 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":340,"threads_started":1}
W20260925 09:18:04.401576 20197 heartbeater.cc:644] Failed to heartbeat to 127.3.19.126:46573 (0 consecutive failures): Network error: Failed to ping master at 127.3.19.126:46573: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
I20260925 09:18:04.402490 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:04.402148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:37335)
0925 09:18:04.402304 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:04.402376 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 09:18:04.402987 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 20069
I20260925 09:18:04.403653 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:04.403206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:39723)
0925 09:18:04.403398 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:04.403484 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 09:18:04.403680 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:37195
--local_ip_for_outbound_sockets=127.3.19.67
--tserver_master_addrs=127.3.19.126:46573
--webserver_port=46051
--webserver_interface=127.3.19.67
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:04.404044 20197 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 09:18:04.738626 20200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:04.738898 20200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:04.739070 20200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:04.746084 20200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:04.746227 20200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:04.746416 20200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:18:04.755678 20200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:37195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=46051
--tserver_master_addrs=127.3.19.126:46573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:04.756906 20200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:04.758179 20200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:04.770856 20205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:04.770913 20208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:04.770890 20206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:04.771726 20200 server_base.cc:1034] running on GCE node
I20260925 09:18:04.772343 20200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:04.773219 20200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:04.774467 20200 hybrid_clock.cc:648] HybridClock initialized: now 1790327884774408 us; error 59 us; skew 500 ppm
I20260925 09:18:04.774763 20200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:04 dist-test-slave-l6qh krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327884, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:04.777818 20200 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:18:04.779072 20200 webserver.cc:460] Webserver started at http://127.3.19.67:46051/ using document root <none> and password file <none>
I20260925 09:18:04.779682 20200 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:04.779783 20200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:04.784778 20200 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 09:18:04.787385 20215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:04.788637 20200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:18:04.788817 20200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "696fe6f9a0eb4ab7b9845853f1903b3e"
format_stamp: "Formatted at 2026-09-25 09:17:19 on dist-test-slave-l6qh"
server_key: "dc77e433ae6a48cd2075028676990ae5"
server_key_iv: "62deb258ab24e48f4e4e8c62b507f6f9"
server_key_version: "encryptionkey@0"
I20260925 09:18:04.789824 20200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:04.808137 20200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:04.812544 20200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:04.812822 20200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:04.813885 20200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:04.815172 20200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:04.815300 20200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:04.815435 20200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:04.815534 20200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:04.848380 20200 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:37195
I20260925 09:18:04.848426 20327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:37195 every 8 connection(s)
I20260925 09:18:04.850000 20200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:18:04.851612 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 20200
I20260925 09:18:04.854736 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:04.852938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:34317)
0925 09:18:04.853731 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:04.853917 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":334,"threads_started":1}
W20260925 09:18:04.855341 20328 heartbeater.cc:644] Failed to heartbeat to 127.3.19.126:46573 (0 consecutive failures): Network error: Failed to ping master at 127.3.19.126:46573: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
I20260925 09:18:04.856065 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:04.855760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:41865)
0925 09:18:04.855897 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:04.855971 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 09:18:04.856788 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:04.856520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:52287)
0925 09:18:04.856636 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:04.856699 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 09:18:04.857008 20328 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 09:18:04.951581 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:04.950772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:52081)
0925 09:18:04.951281 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:04.951367 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":260,"threads_started":1}
I20260925 09:18:05.405428 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:05.404528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:36251)
0925 09:18:05.405016 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:05.405115 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":268,"threads_started":1}
I20260925 09:18:05.858291 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:05.857580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:56933)
0925 09:18:05.857964 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:05.858055 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":191,"threads_started":1}
I20260925 09:18:05.953059 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:05.952407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:47641)
0925 09:18:05.952830 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:05.952903 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":223,"threads_started":1}
I20260925 09:18:06.406899 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:06.406263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:37091)
0925 09:18:06.406674 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:06.406746 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":205,"threads_started":1}
I20260925 09:18:06.860080 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:06.859399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:33651)
0925 09:18:06.859842 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:06.859925 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":212,"threads_started":1}
I20260925 09:18:06.954769 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:06.954020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:53563)
0925 09:18:06.954505 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:06.954600 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":252,"threads_started":1}
I20260925 09:18:07.408514 20339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:07.407809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:45701)
0925 09:18:07.408259 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:07.408341 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":252,"threads_started":1}
I20260925 09:18:07.861855 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:07.861056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:41379)
0925 09:18:07.861559 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:07.861686 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":293,"threads_started":1}
I20260925 09:18:07.956429 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:07.955739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:52063)
0925 09:18:07.956183 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:07.956265 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":227,"threads_started":1}
I20260925 09:18:08.410274 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:08.409525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:47709)
0925 09:18:08.409996 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:08.410082 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":234,"threads_started":1}
I20260925 09:18:08.863356 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:08.862650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:33965)
0925 09:18:08.863097 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:08.863176 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":204,"threads_started":1}
I20260925 09:18:08.958076 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:08.957361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:58451)
0925 09:18:08.957861 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:08.957931 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":286,"threads_started":1}
I20260925 09:18:09.411880 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:09.411090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:33309)
0925 09:18:09.411641 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:09.411718 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":327,"threads_started":1}
I20260925 09:18:09.864917 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:09.864261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:43521)
0925 09:18:09.864694 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:09.864767 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":225,"threads_started":1}
I20260925 09:18:09.959626 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:09.958945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:54801)
0925 09:18:09.959366 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:09.959457 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":221,"threads_started":1}
I20260925 09:18:10.413482 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:10.412694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:35387)
0925 09:18:10.413166 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:10.413324 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":267,"threads_started":1}
I20260925 09:18:10.866466 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:10.865802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:53837)
0925 09:18:10.866238 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:10.866317 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":229,"threads_started":1}
I20260925 09:18:10.961355 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:10.960546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:50741)
0925 09:18:10.961003 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:10.961078 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":215,"threads_started":1}
I20260925 09:18:11.415488 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:11.414392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:57373)
0925 09:18:11.415044 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:11.415158 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":293,"threads_started":1}
I20260925 09:18:11.868064 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:11.867305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:43251)
0925 09:18:11.867790 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:11.867861 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":207,"threads_started":1}
I20260925 09:18:11.962872 20353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:11.962217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:48289)
0925 09:18:11.962629 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:11.962706 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":227,"threads_started":1}
I20260925 09:18:12.417034 20354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:12.416345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:48761)
0925 09:18:12.416794 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:12.416871 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":232,"threads_started":1}
I20260925 09:18:12.869678 20355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:12.868979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:39375)
0925 09:18:12.869427 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:12.869505 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":250,"threads_started":1}
I20260925 09:18:12.964526 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:12.963816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:33903)
0925 09:18:12.964270 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:12.964366 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":226,"threads_started":1}
I20260925 09:18:13.418538 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:13.417850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:52611)
0925 09:18:13.418296 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:13.418371 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":219,"threads_started":1}
I20260925 09:18:13.871202 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:13.870531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:38749)
0925 09:18:13.870976 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:13.871047 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":226,"threads_started":1}
I20260925 09:18:13.966197 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:13.965543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:48097)
0925 09:18:13.965957 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:13.966028 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":238,"threads_started":1}
I20260925 09:18:14.420075 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:14.419422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:36377)
0925 09:18:14.419845 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:14.419919 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":238,"threads_started":1}
I20260925 09:18:14.872946 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:14.872087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:47487)
0925 09:18:14.872627 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:14.872750 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
I20260925 09:18:14.967608 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:14.966951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:52175)
0925 09:18:14.967377 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:14.967452 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":240,"threads_started":1}
I20260925 09:18:15.421674 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:15.420923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:44067)
0925 09:18:15.421413 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:15.421500 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":221,"threads_started":1}
I20260925 09:18:15.874681 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:15.873863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:44323)
0925 09:18:15.874381 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:15.874469 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":229,"threads_started":1}
I20260925 09:18:15.969290 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:15.968527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:54883)
0925 09:18:15.968957 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:15.969041 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":234,"threads_started":1}
I20260925 09:18:16.423130 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:16.422517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:51831)
0925 09:18:16.422916 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:16.422989 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":205,"threads_started":1}
I20260925 09:18:16.876184 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:16.875506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:37443)
0925 09:18:16.875955 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:16.876030 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":250,"threads_started":1}
I20260925 09:18:16.970894 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:16.970176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:58615)
0925 09:18:16.970608 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:16.970730 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":213,"threads_started":1}
I20260925 09:18:17.424647 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:17.423935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:38995)
0925 09:18:17.424379 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:17.424470 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":212,"threads_started":1}
I20260925 09:18:17.877799 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:17.877008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:52761)
0925 09:18:17.877468 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:17.877600 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":263,"threads_started":1}
I20260925 09:18:17.972538 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:17.971798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:52515)
0925 09:18:17.972277 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:17.972355 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":254,"threads_started":1}
I20260925 09:18:18.426209 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:18.425507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:33833)
0925 09:18:18.425947 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:18.426024 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":228,"threads_started":1}
I20260925 09:18:18.879424 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:18.878702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:44187)
0925 09:18:18.879159 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:18.879244 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":226,"threads_started":1}
I20260925 09:18:18.974407 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:18.973751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.65:35339)
0925 09:18:18.974167 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:18.974248 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":213,"threads_started":1}
I20260925 09:18:19.427964 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:19.427271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.66:40805)
0925 09:18:19.427692 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:19.427785 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":211,"threads_started":1}
I20260925 09:18:19.855816 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 19897
I20260925 09:18:19.877707 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:19.856785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.3.19.65:37555)
0925 09:18:19.857392 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:19.857430 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:19.857477 (+ 47us) server_negotiation.cc:407] Connection header received
0925 09:18:19.859418 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:19.859428 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:19.859851 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:19.860216 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:19.861777 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:19.863059 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:19.864332 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:19.864985 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:19.865184 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:19.871648 (+ 6464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:19.871656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:19.871672 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:19.871707 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:19.874451 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:19.875135 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:19.875143 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:19.875149 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:19.875312 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:19.875688 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:19.875694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:19.875698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:19.876204 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:19.876518 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:19.876942 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 09:18:19.877400 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":280,"threads_started":1}
I20260925 09:18:19.879108 20061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:19.881418 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:19.880572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:46573 (local address 127.3.19.67:44703)
0925 09:18:19.881051 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:19.881180 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.126:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":216,"threads_started":1}
I20260925 09:18:19.964360 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 19897
I20260925 09:18:19.983086 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 20069
I20260925 09:18:20.000350 20379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:19.983992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.3.19.66:37315)
0925 09:18:19.984505 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:19.984520 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:19.984558 (+ 38us) server_negotiation.cc:407] Connection header received
0925 09:18:19.986198 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:19.986207 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:19.986641 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:19.986932 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:19.988246 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:19.989567 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:19.990751 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:19.991377 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:19.991509 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:19.994532 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:19.994539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:19.994554 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:19.994589 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:19.997601 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:19.998167 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:19.998175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:19.998181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:19.998335 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:19.998694 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:19.998697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:19.998699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:19.999168 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:19.999397 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:19.999757 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 09:18:20.000064 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":243,"threads_started":1}
I20260925 09:18:20.001768 20192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:20.083961 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 20069
I20260925 09:18:20.101946 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 20200
I20260925 09:18:20.119746 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:20.102983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.3.19.67:37195)
0925 09:18:20.103542 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:20.103555 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:20.103595 (+ 40us) server_negotiation.cc:407] Connection header received
0925 09:18:20.105218 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:20.105278 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:20.105741 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:20.106092 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:20.107521 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:20.108751 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:20.110136 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:20.110808 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:20.111032 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:20.113592 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:20.113606 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:20.113620 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:20.113654 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:20.116444 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:20.117077 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:20.117092 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:20.117100 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:20.117438 (+ 338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:20.117828 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:20.117836 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:20.117840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:20.118320 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:20.118632 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:20.119024 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 09:18:20.119432 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":259,"threads_started":1}
I20260925 09:18:20.121098 20323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:20.203106 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 20200
2026-09-25T09:18:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64254 ms)
[----------] 30 tests from SecurityITest (299710 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:18:20 dist-test-slave-l6qh krb5kdc[20385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:18:20 dist-test-slave-l6qh krb5kdc[20385](info): set up 2 sockets
Sep 25 09:18:20 dist-test-slave-l6qh krb5kdc[20385](info): commencing operation
krb5kdc: starting...
W20260925 09:18:21.314405 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:18:21 dist-test-slave-l6qh krb5kdc[20385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:18:21.598852 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33055
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:33055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:21.934386 20398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:21.934762 20398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:21.934870 20398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:21.942075 20398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:21.942201 20398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:21.942248 20398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:21.942306 20398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:21.942368 20398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:21.952034 20398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:33055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:21.953286 20398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:21.954505 20398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:21.967070 20403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:21.967038 20404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:21.966984 20406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:21.967494 20398 server_base.cc:1034] running on GCE node
I20260925 09:18:21.968655 20398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:21.970144 20398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:21.971377 20398 hybrid_clock.cc:648] HybridClock initialized: now 1790327901971325 us; error 42 us; skew 500 ppm
I20260925 09:18:21.971737 20398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:21 dist-test-slave-l6qh krb5kdc[20385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327901, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:21.974954 20398 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:18:21.976469 20398 webserver.cc:460] Webserver started at http://127.3.19.126:37089/ using document root <none> and password file <none>
I20260925 09:18:21.977149 20398 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:21.977312 20398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:21.977630 20398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:21.980592 20398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "241066d46f1e45aba9055930bd614e93"
format_stamp: "Formatted at 2026-09-25 09:18:21 on dist-test-slave-l6qh"
server_key: "bfa95b675264d4f7dfcb09feb8afc63f"
server_key_iv: "14d8723291a217f43e4de9aabb9e95f6"
server_key_version: "encryptionkey@0"
I20260925 09:18:21.981408 20398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "241066d46f1e45aba9055930bd614e93"
format_stamp: "Formatted at 2026-09-25 09:18:21 on dist-test-slave-l6qh"
server_key: "bfa95b675264d4f7dfcb09feb8afc63f"
server_key_iv: "14d8723291a217f43e4de9aabb9e95f6"
server_key_version: "encryptionkey@0"
I20260925 09:18:21.986486 20398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 09:18:21.990181 20413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:21.991494 20398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:18:21.991727 20398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "241066d46f1e45aba9055930bd614e93"
format_stamp: "Formatted at 2026-09-25 09:18:21 on dist-test-slave-l6qh"
server_key: "bfa95b675264d4f7dfcb09feb8afc63f"
server_key_iv: "14d8723291a217f43e4de9aabb9e95f6"
server_key_version: "encryptionkey@0"
I20260925 09:18:21.991943 20398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:22.014065 20398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:22.018162 20398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:22.018461 20398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:22.037266 20398 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:33055
I20260925 09:18:22.037362 20464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:33055 every 8 connection(s)
I20260925 09:18:22.038643 20398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:22.042542 20465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:22.044741 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 20398
I20260925 09:18:22.044950 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:22.045797 3149 external_mini_cluster.cc:1428] Setting key 9583714d784efeddf5e123d49285ec15
I20260925 09:18:22.055658 20465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93: Bootstrap starting.
I20260925 09:18:22.059608 20465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:22.060880 20465 log.cc:824] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93: Log is configured to *not* fsync() on all Append() calls
Sep 25 09:18:22 dist-test-slave-l6qh krb5kdc[20385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:18:22.064085 20465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93: No bootstrap required, opened a new log
I20260925 09:18:22.070103 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:22.047911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.3.19.126:33055)
0925 09:18:22.048819 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:22.048830 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:22.048875 (+ 45us) server_negotiation.cc:407] Connection header received
0925 09:18:22.050583 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:22.050605 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:22.051002 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:22.051324 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:22.053137 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:22.054558 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:22.056458 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:22.057125 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:22.057320 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:22.062171 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:22.062178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:22.062193 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:22.062232 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:22.066034 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:22.066722 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:22.066739 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:22.066756 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:22.066924 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:22.067323 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:22.067329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:22.067332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:22.067847 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:22.068172 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:22.068617 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 09:18:22.069055 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":399,"threads_started":1}
I20260925 09:18:22.072211 20465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241066d46f1e45aba9055930bd614e93" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33055 } }
I20260925 09:18:22.072603 20465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:22.072746 20465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241066d46f1e45aba9055930bd614e93, State: Initialized, Role: FOLLOWER
I20260925 09:18:22.073361 20465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241066d46f1e45aba9055930bd614e93" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33055 } }
I20260925 09:18:22.073601 20465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:22.073741 20465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:22.073921 20465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:22.076056 20465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241066d46f1e45aba9055930bd614e93" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33055 } }
I20260925 09:18:22.076530 20465 leader_election.cc:302] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 241066d46f1e45aba9055930bd614e93; no voters:
I20260925 09:18:22.077143 20465 leader_election.cc:288] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:22.077276 20470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:22.078030 20470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [term 1 LEADER]: Becoming Leader. State: Replica: 241066d46f1e45aba9055930bd614e93, State: Running, Role: LEADER
I20260925 09:18:22.078642 20470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241066d46f1e45aba9055930bd614e93" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33055 } }
I20260925 09:18:22.079471 20465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:22.081777 20472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 241066d46f1e45aba9055930bd614e93. Latest consensus state: current_term: 1 leader_uuid: "241066d46f1e45aba9055930bd614e93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241066d46f1e45aba9055930bd614e93" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33055 } } }
I20260925 09:18:22.082242 20472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:22.083050 20471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "241066d46f1e45aba9055930bd614e93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241066d46f1e45aba9055930bd614e93" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33055 } } }
I20260925 09:18:22.083410 20471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:22.086586 20477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:22.091796 20477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:22.100756 20477 catalog_manager.cc:1347] Generated new cluster ID: a320dc926ce64f348d84156adeb8b711
I20260925 09:18:22.100871 20477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:22.132869 20477 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:22.134284 20477 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:22.142650 20477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93: Generated new TSK 0
I20260925 09:18:22.143967 20477 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:18:22.218832 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33055
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:22.563218 20491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:22.563478 20491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:22.563570 20491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:22.570750 20491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:22.570858 20491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:22.570968 20491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:18:22.580813 20491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:22.582039 20491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:22.583458 20491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:22.596719 20496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:22.596750 20499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:22.596727 20497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:22.597747 20491 server_base.cc:1034] running on GCE node
I20260925 09:18:22.598332 20491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:22.599143 20491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:22.600368 20491 hybrid_clock.cc:648] HybridClock initialized: now 1790327902600310 us; error 66 us; skew 500 ppm
I20260925 09:18:22.600684 20491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:22 dist-test-slave-l6qh krb5kdc[20385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327902, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:22.603742 20491 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:18:22.605135 20491 webserver.cc:460] Webserver started at http://127.3.19.65:36929/ using document root <none> and password file <none>
I20260925 09:18:22.605826 20491 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:22.605973 20491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:22.606273 20491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:22.609009 20491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e"
format_stamp: "Formatted at 2026-09-25 09:18:22 on dist-test-slave-l6qh"
server_key: "fc863291206916aeec7dfe22d30d402a"
server_key_iv: "d6e0a011ca0d5e3cd6b9eaa3b803a42e"
server_key_version: "encryptionkey@0"
I20260925 09:18:22.609858 20491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e"
format_stamp: "Formatted at 2026-09-25 09:18:22 on dist-test-slave-l6qh"
server_key: "fc863291206916aeec7dfe22d30d402a"
server_key_iv: "d6e0a011ca0d5e3cd6b9eaa3b803a42e"
server_key_version: "encryptionkey@0"
I20260925 09:18:22.614892 20491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:18:22.618384 20506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:22.619683 20491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 09:18:22.619853 20491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e"
format_stamp: "Formatted at 2026-09-25 09:18:22 on dist-test-slave-l6qh"
server_key: "fc863291206916aeec7dfe22d30d402a"
server_key_iv: "d6e0a011ca0d5e3cd6b9eaa3b803a42e"
server_key_version: "encryptionkey@0"
I20260925 09:18:22.620057 20491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:22.643220 20491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:22.647075 20491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:22.647367 20491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:22.648360 20491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:22.649641 20491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:22.649741 20491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:22.649876 20491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:22.649964 20491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:22.683360 20491 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:41321
I20260925 09:18:22.683441 20618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:41321 every 8 connection(s)
I20260925 09:18:22.684649 20491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:18:22.688747 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 20491
I20260925 09:18:22.688952 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:18:22.689800 3149 external_mini_cluster.cc:1428] Setting key d6ac18bb0a433c84c657d408f9276a00
Sep 25 09:18:22 dist-test-slave-l6qh krb5kdc[20385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327902, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:18:22.707216 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:22.687381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:57261 (local address 127.3.19.126:33055)
0925 09:18:22.687860 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:22.687866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:22.689286 (+ 1420us) server_negotiation.cc:407] Connection header received
0925 09:18:22.691447 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:22.691453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:22.691548 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:22.691702 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:22.693678 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:22.694395 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:22.696849 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:22.697369 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:22.697466 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:22.701620 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:22.701624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:22.701628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:22.701658 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:22.704192 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:22.704893 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:22.704903 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:22.704905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:22.705018 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:22.705513 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:22.705518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:22.705521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:22.705787 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:22.705939 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:22.706639 (+ 700us) server_negotiation.cc:299] Negotiation successful
0925 09:18:22.706859 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":226,"threads_started":1}
I20260925 09:18:22.708175 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:22.687977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33055 (local address 127.3.19.65:57261)
0925 09:18:22.688890 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:22.688945 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:22.691190 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:22.691954 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:22.691965 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:22.692678 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:22.693506 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:22.693533 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:22.694559 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:22.694567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:22.696694 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:22.696710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:22.697586 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:22.697596 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:22.697921 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:22.698620 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:18:22.698653 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:22.701399 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:22.704333 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:22.704341 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:22.704356 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:22.704764 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:22.705131 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:22.705137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:22.705141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:22.705399 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:22.705919 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:22.705926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:22.706270 (+ 344us) client_negotiation.cc:769] Sending connection context
0925 09:18:22.706517 (+ 247us) client_negotiation.cc:240] Negotiation successful
0925 09:18:22.706992 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":355,"threads_started":1}
I20260925 09:18:22.709815 20619 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33055
I20260925 09:18:22.710153 20619 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:22.710940 20619 heartbeater.cc:511] Master 127.3.19.126:33055 requested a full tablet report, sending...
I20260925 09:18:22.713346 20430 ts_manager.cc:194] Registered new tserver with Master: 46031b71cdcc4e92bfc1947a8ecd7d0e (127.3.19.65:41321)
I20260925 09:18:22.715615 20430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:57261
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:18:22.748786 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33055
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:23.068452 20625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:23.068706 20625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:23.068852 20625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:23.075915 20625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:23.076006 20625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:23.076182 20625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:18:23.085428 20625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:23.086642 20625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:23.087918 20625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:23.100092 20630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:23.100133 20631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:23.100191 20633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:23.100870 20625 server_base.cc:1034] running on GCE node
I20260925 09:18:23.101495 20625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:23.102350 20625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:23.103578 20625 hybrid_clock.cc:648] HybridClock initialized: now 1790327903103533 us; error 43 us; skew 500 ppm
I20260925 09:18:23.103883 20625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:23 dist-test-slave-l6qh krb5kdc[20385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327903, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:23.106949 20625 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:18:23.108354 20625 webserver.cc:460] Webserver started at http://127.3.19.66:45015/ using document root <none> and password file <none>
I20260925 09:18:23.108981 20625 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:23.109119 20625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:23.109473 20625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:23.112339 20625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb"
format_stamp: "Formatted at 2026-09-25 09:18:23 on dist-test-slave-l6qh"
server_key: "fed00480c036c46a4810ea5eb591e1e5"
server_key_iv: "7838f16ca811511607bd23ee7d2ea99e"
server_key_version: "encryptionkey@0"
I20260925 09:18:23.113103 20625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb"
format_stamp: "Formatted at 2026-09-25 09:18:23 on dist-test-slave-l6qh"
server_key: "fed00480c036c46a4810ea5eb591e1e5"
server_key_iv: "7838f16ca811511607bd23ee7d2ea99e"
server_key_version: "encryptionkey@0"
I20260925 09:18:23.118084 20625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 09:18:23.121429 20640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:23.122812 20625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 09:18:23.123025 20625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb"
format_stamp: "Formatted at 2026-09-25 09:18:23 on dist-test-slave-l6qh"
server_key: "fed00480c036c46a4810ea5eb591e1e5"
server_key_iv: "7838f16ca811511607bd23ee7d2ea99e"
server_key_version: "encryptionkey@0"
I20260925 09:18:23.123234 20625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:23.152356 20625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:23.156224 20625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:23.156502 20625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:23.157416 20625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:23.158641 20625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:23.158751 20625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:23.158908 20625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:23.158978 20625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:23.191927 20625 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:33153
I20260925 09:18:23.191979 20752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:33153 every 8 connection(s)
I20260925 09:18:23.193147 20625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:18:23.201805 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 20625
I20260925 09:18:23.202105 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:18:23.202658 3149 external_mini_cluster.cc:1428] Setting key d4fa2eaaea1cee40623ac0749fbbcbcf
Sep 25 09:18:23 dist-test-slave-l6qh krb5kdc[20385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327903, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:18:23.215662 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:23.196166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:57283 (local address 127.3.19.126:33055)
0925 09:18:23.196358 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:23.196362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:23.197645 (+ 1283us) server_negotiation.cc:407] Connection header received
0925 09:18:23.199390 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:23.199395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:23.199463 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:23.199552 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:23.200997 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.201756 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.203797 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.204454 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.204591 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:23.208764 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:23.208773 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:23.208781 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:23.208825 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:23.212457 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:23.213130 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:23.213137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:23.213143 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:23.213330 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:23.213740 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:23.213765 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:23.213771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:23.214147 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:23.214302 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:23.214953 (+ 651us) server_negotiation.cc:299] Negotiation successful
0925 09:18:23.215222 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 09:18:23.216552 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:23.196680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33055 (local address 127.3.19.66:57283)
0925 09:18:23.197461 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:23.197498 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:23.199121 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:23.199770 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:23.199777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:23.200260 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:23.200856 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:23.200873 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.201938 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.201945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:23.203666 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:23.203678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.204742 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.204750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:23.205057 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:23.205800 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:18:23.205833 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:23.208496 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:23.212593 (+ 4097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:23.212600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:23.212612 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:23.213010 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:23.213443 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:23.213446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:23.213449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:23.213621 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:23.214270 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:23.214275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:23.214571 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 09:18:23.214809 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 09:18:23.215192 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":280,"threads_started":1}
I20260925 09:18:23.218079 20753 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33055
I20260925 09:18:23.218398 20753 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:23.219102 20753 heartbeater.cc:511] Master 127.3.19.126:33055 requested a full tablet report, sending...
I20260925 09:18:23.220598 20430 ts_manager.cc:194] Registered new tserver with Master: c0ba8d1b46f243e0b7457bf76ae99bcb (127.3.19.66:33153)
I20260925 09:18:23.221477 20430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:57283
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:18:23.260212 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33055
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:23.580624 20758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:23.580947 20758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:23.581153 20758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:23.588747 20758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:23.588881 20758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:23.589071 20758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:18:23.599169 20758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:23.600440 20758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:23.601816 20758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:23.614923 20766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:23.614950 20763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:23.614857 20764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:23.615942 20758 server_base.cc:1034] running on GCE node
I20260925 09:18:23.616524 20758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:23.617525 20758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:23.618749 20758 hybrid_clock.cc:648] HybridClock initialized: now 1790327903618698 us; error 43 us; skew 500 ppm
I20260925 09:18:23.619069 20758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:23 dist-test-slave-l6qh krb5kdc[20385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327903, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:23.622287 20758 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:18:23.623680 20758 webserver.cc:460] Webserver started at http://127.3.19.67:33815/ using document root <none> and password file <none>
I20260925 09:18:23.624366 20758 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:23.624461 20758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:23.624770 20758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:23.627712 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "e133b1e4215b436c81f53688f1646649"
format_stamp: "Formatted at 2026-09-25 09:18:23 on dist-test-slave-l6qh"
server_key: "b3449f197f10c2b065fb7c09451342c3"
server_key_iv: "90f555b851cbd91446ed7c33561335bc"
server_key_version: "encryptionkey@0"
I20260925 09:18:23.628450 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "e133b1e4215b436c81f53688f1646649"
format_stamp: "Formatted at 2026-09-25 09:18:23 on dist-test-slave-l6qh"
server_key: "b3449f197f10c2b065fb7c09451342c3"
server_key_iv: "90f555b851cbd91446ed7c33561335bc"
server_key_version: "encryptionkey@0"
I20260925 09:18:23.633603 20758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 09:18:23.637138 20773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:23.638600 20758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:18:23.638773 20758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "e133b1e4215b436c81f53688f1646649"
format_stamp: "Formatted at 2026-09-25 09:18:23 on dist-test-slave-l6qh"
server_key: "b3449f197f10c2b065fb7c09451342c3"
server_key_iv: "90f555b851cbd91446ed7c33561335bc"
server_key_version: "encryptionkey@0"
I20260925 09:18:23.638984 20758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:23.669833 20758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:23.674422 20758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:23.674732 20758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:23.675755 20758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:23.677215 20758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:23.677349 20758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 09:18:23.677485 20758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:23.677542 20758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:23.712189 20758 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:41895
I20260925 09:18:23.712263 20885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:41895 every 8 connection(s)
I20260925 09:18:23.713481 20758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:18:23.719072 20619 heartbeater.cc:503] Master 127.3.19.126:33055 was elected leader, sending a full tablet report...
I20260925 09:18:23.722224 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 20758
I20260925 09:18:23.722407 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:18:23.723067 3149 external_mini_cluster.cc:1428] Setting key 996eb533553ae89a4fd156236f3968e9
Sep 25 09:18:23 dist-test-slave-l6qh krb5kdc[20385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327903, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:18:23.734803 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:23.716095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:35389 (local address 127.3.19.126:33055)
0925 09:18:23.716599 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:23.716603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:23.717626 (+ 1023us) server_negotiation.cc:407] Connection header received
0925 09:18:23.719662 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:23.719674 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:23.719757 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:23.719878 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:23.721715 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.722579 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.724741 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.725390 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.725529 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:23.729192 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:23.729196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:23.729199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:23.729289 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:23.731781 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:23.732474 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:23.732480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:23.732483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:23.732638 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:23.733054 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:23.733060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:23.733064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:23.733393 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:23.733551 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:23.734181 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 09:18:23.734409 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":300,"threads_started":1}
I20260925 09:18:23.735661 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:23.716637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33055 (local address 127.3.19.67:35389)
0925 09:18:23.717428 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:23.717493 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:23.719245 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:23.720139 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:23.720150 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:23.720867 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:23.721555 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:23.721582 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.722779 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.722785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:23.724584 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:23.724605 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.725611 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.725621 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:23.725862 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:23.726695 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:18:23.726716 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:23.728987 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:23.731920 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:23.731929 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:23.731945 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:23.732355 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:23.732749 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:23.732755 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:23.732759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:23.732941 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:23.733516 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:23.733524 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:23.733838 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 09:18:23.734090 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 09:18:23.734592 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":253,"threads_started":1}
I20260925 09:18:23.736917 20886 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33055
I20260925 09:18:23.737310 20886 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:23.738099 20886 heartbeater.cc:511] Master 127.3.19.126:33055 requested a full tablet report, sending...
I20260925 09:18:23.739508 20430 ts_manager.cc:194] Registered new tserver with Master: e133b1e4215b436c81f53688f1646649 (127.3.19.67:41895)
I20260925 09:18:23.740317 20430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:35389
I20260925 09:18:23.751264 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:18:23.768622 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:23.755735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.3.19.126:33055)
0925 09:18:23.755951 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:23.755960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:23.756291 (+ 331us) server_negotiation.cc:407] Connection header received
0925 09:18:23.756416 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:23.756420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:23.756504 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:23.756582 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:23.758025 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.758737 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.760097 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.760470 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.760570 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:23.762972 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:23.762976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:23.762979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:23.763008 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:23.765966 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:23.766691 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:23.766695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:23.766697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:23.766841 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:23.767158 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:23.767162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:23.767164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:23.767517 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:23.767741 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:23.768025 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 09:18:23.768204 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 09:18:23.772578 20430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:18:23.779527 20430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:18:23.804455 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:23.791369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.3.19.66:33153)
0925 09:18:23.793009 (+ 1640us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:23.793017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:23.793044 (+ 27us) server_negotiation.cc:407] Connection header received
0925 09:18:23.793137 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:23.793146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:23.793481 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:23.793687 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:23.795390 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.797489 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.800970 (+ 3481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.803023 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.803206 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:23.803467 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:23.803981 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 09:18:23.804155 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1488,"thread_start_us":1568,"threads_started":1}
I20260925 09:18:23.804533 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:23.791148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:33153 (local address 127.0.0.1:53034)
0925 09:18:23.792154 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:23.792203 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:23.792419 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:23.793708 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:23.793715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:23.793748 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:23.795098 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:23.795115 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.797704 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.797713 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:23.800761 (+ 3048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:23.800783 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.803253 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.803263 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:23.803561 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:23.803644 (+ 83us) client_negotiation.cc:769] Sending connection context
0925 09:18:23.803802 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 09:18:23.804053 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":2103,"threads_started":1}
I20260925 09:18:23.808887 20688 tablet_service.cc:1467] Processing CreateTablet for tablet 0252ed18d0bd4b4e984ef16d6bb93f61 (DEFAULT_TABLE table=test-table [id=f3c513e531324f63be5920d46b1112e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:23.810563 20688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0252ed18d0bd4b4e984ef16d6bb93f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:23.813572 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:23.792084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.3.19.67:41895)
0925 09:18:23.805067 (+ 12983us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:23.805075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:23.805103 (+ 28us) server_negotiation.cc:407] Connection header received
0925 09:18:23.805204 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:23.805211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:23.805428 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:23.806934 (+ 1506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:23.807178 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.808336 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.810925 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.812265 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.812718 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:23.812928 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:23.813052 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 09:18:23.813205 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12818,"thread_start_us":12560,"threads_started":1}
I20260925 09:18:23.813591 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:23.791374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:41895 (local address 127.0.0.1:54444)
0925 09:18:23.793555 (+ 2181us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:23.793575 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:23.793679 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:23.805646 (+ 11967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:23.805650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:23.805674 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:23.806730 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:23.806741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.808530 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.808538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:23.810757 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:23.810772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.812495 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.812501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:23.812809 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:23.812898 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 09:18:23.813008 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 09:18:23.813169 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2060,"mutex_wait_us":109,"thread_start_us":1056,"threads_started":1}
I20260925 09:18:23.813977 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:23.793700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36674 (local address 127.3.19.65:41321)
0925 09:18:23.803981 (+ 10281us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:23.803990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:23.804016 (+ 26us) server_negotiation.cc:407] Connection header received
0925 09:18:23.804104 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:23.804111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:23.804376 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:23.805350 (+ 974us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:23.805543 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.806939 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.809845 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.812211 (+ 2366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.812361 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:23.812598 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:23.813171 (+ 573us) server_negotiation.cc:299] Negotiation successful
0925 09:18:23.813398 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10116,"thread_start_us":9983,"threads_started":1}
I20260925 09:18:23.815249 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:23.793901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:41321 (local address 127.0.0.1:36674)
0925 09:18:23.794484 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:23.794510 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:23.794656 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:23.804603 (+ 9947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:23.804610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:23.804634 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:23.805128 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:23.805136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.807081 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.807086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:23.809681 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:23.809700 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:23.812463 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:23.812476 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:23.812808 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:23.812907 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 09:18:23.813033 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 09:18:23.814881 (+ 1848us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":1741,"thread_start_us":340,"threads_started":1}
I20260925 09:18:23.817556 20821 tablet_service.cc:1467] Processing CreateTablet for tablet 0252ed18d0bd4b4e984ef16d6bb93f61 (DEFAULT_TABLE table=test-table [id=f3c513e531324f63be5920d46b1112e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:23.818771 20821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0252ed18d0bd4b4e984ef16d6bb93f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:23.819331 20554 tablet_service.cc:1467] Processing CreateTablet for tablet 0252ed18d0bd4b4e984ef16d6bb93f61 (DEFAULT_TABLE table=test-table [id=f3c513e531324f63be5920d46b1112e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:23.820976 20554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0252ed18d0bd4b4e984ef16d6bb93f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:23.826719 20905 tablet_bootstrap.cc:492] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb: Bootstrap starting.
I20260925 09:18:23.830384 20905 tablet_bootstrap.cc:654] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:23.831238 20906 tablet_bootstrap.cc:492] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649: Bootstrap starting.
I20260925 09:18:23.831835 20905 log.cc:824] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:23.834986 20905 tablet_bootstrap.cc:492] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb: No bootstrap required, opened a new log
I20260925 09:18:23.835264 20905 ts_tablet_manager.cc:1397] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 09:18:23.835244 20906 tablet_bootstrap.cc:654] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:23.836743 20906 log.cc:824] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:23.837204 20908 tablet_bootstrap.cc:492] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e: Bootstrap starting.
I20260925 09:18:23.840952 20908 tablet_bootstrap.cc:654] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:23.842218 20908 log.cc:824] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:23.844082 20906 tablet_bootstrap.cc:492] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649: No bootstrap required, opened a new log
I20260925 09:18:23.844388 20906 ts_tablet_manager.cc:1397] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260925 09:18:23.845640 20905 raft_consensus.cc:348] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33153 } } peers { permanent_uuid: "e133b1e4215b436c81f53688f1646649" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41895 } } peers { permanent_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 41321 } }
I20260925 09:18:23.845881 20908 tablet_bootstrap.cc:492] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e: No bootstrap required, opened a new log
I20260925 09:18:23.846031 20905 raft_consensus.cc:374] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:23.846155 20905 raft_consensus.cc:729] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ba8d1b46f243e0b7457bf76ae99bcb, State: Initialized, Role: FOLLOWER
I20260925 09:18:23.846196 20908 ts_tablet_manager.cc:1397] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 09:18:23.846766 20905 consensus_queue.cc:260] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33153 } } peers { permanent_uuid: "e133b1e4215b436c81f53688f1646649" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41895 } } peers { permanent_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 41321 } }
I20260925 09:18:23.847872 20753 heartbeater.cc:503] Master 127.3.19.126:33055 was elected leader, sending a full tablet report...
I20260925 09:18:23.848366 20905 ts_tablet_manager.cc:1428] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260925 09:18:23.858718 20908 raft_consensus.cc:348] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33153 } } peers { permanent_uuid: "e133b1e4215b436c81f53688f1646649" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41895 } } peers { permanent_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 41321 } }
I20260925 09:18:23.858712 20906 raft_consensus.cc:348] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33153 } } peers { permanent_uuid: "e133b1e4215b436c81f53688f1646649" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41895 } } peers { permanent_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 41321 } }
I20260925 09:18:23.859231 20906 raft_consensus.cc:374] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:23.859230 20908 raft_consensus.cc:374] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:23.859385 20906 raft_consensus.cc:729] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e133b1e4215b436c81f53688f1646649, State: Initialized, Role: FOLLOWER
I20260925 09:18:23.859386 20908 raft_consensus.cc:729] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46031b71cdcc4e92bfc1947a8ecd7d0e, State: Initialized, Role: FOLLOWER
I20260925 09:18:23.860108 20906 consensus_queue.cc:260] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33153 } } peers { permanent_uuid: "e133b1e4215b436c81f53688f1646649" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41895 } } peers { permanent_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 41321 } }
I20260925 09:18:23.860100 20908 consensus_queue.cc:260] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33153 } } peers { permanent_uuid: "e133b1e4215b436c81f53688f1646649" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41895 } } peers { permanent_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 41321 } }
I20260925 09:18:23.861733 20886 heartbeater.cc:503] Master 127.3.19.126:33055 was elected leader, sending a full tablet report...
I20260925 09:18:23.862254 20908 ts_tablet_manager.cc:1428] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260925 09:18:23.863525 20906 ts_tablet_manager.cc:1428] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649: Time spent starting tablet: real 0.019s user 0.016s sys 0.002s
W20260925 09:18:23.937196 20620 tablet.cc:2369] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:18:23.945113 20754 tablet.cc:2369] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:18:23.965154 20887 tablet.cc:2369] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:18:24.164153 20913 raft_consensus.cc:482] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:18:24.164482 20913 raft_consensus.cc:504] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33153 } } peers { permanent_uuid: "e133b1e4215b436c81f53688f1646649" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41895 } } peers { permanent_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 41321 } }
I20260925 09:18:24.166260 20913 leader_election.cc:288] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ba8d1b46f243e0b7457bf76ae99bcb (127.3.19.66:33153), e133b1e4215b436c81f53688f1646649 (127.3.19.67:41895)
I20260925 09:18:24.172928 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:24.166658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:34487 (local address 127.3.19.66:33153)
0925 09:18:24.166880 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:24.166889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:24.167684 (+ 795us) server_negotiation.cc:407] Connection header received
0925 09:18:24.167870 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:24.167874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:24.167930 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:24.168024 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:24.168676 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.169437 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.171175 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.171953 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.172112 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:24.172220 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:24.172513 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 09:18:24.172637 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 09:18:24.172943 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:24.166710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:54371 (local address 127.3.19.67:41895)
0925 09:18:24.166907 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:24.166914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:24.167586 (+ 672us) server_negotiation.cc:407] Connection header received
0925 09:18:24.167802 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:24.167806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:24.167864 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:24.167954 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:24.168553 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.169331 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.171190 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.171969 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.172120 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:24.172238 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:24.172548 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 09:18:24.172689 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 09:18:24.173422 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:24.166645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:41895 (local address 127.3.19.65:54371)
0925 09:18:24.167474 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:24.167489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:24.167658 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:24.167981 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:24.167985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:24.168009 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:24.168383 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:24.168395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.169452 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.169459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:24.171030 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:24.171042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.172252 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.172261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:24.172394 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:24.172416 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 09:18:24.172521 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 09:18:24.172651 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":3,"thread_start_us":476,"threads_started":1}
I20260925 09:18:24.173454 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:24.166649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:33153 (local address 127.3.19.65:34487)
0925 09:18:24.167564 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:24.167577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:24.167723 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:24.168036 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:24.168040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:24.168067 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:24.168495 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:24.168503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.169572 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.169577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:24.171020 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:24.171031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.172206 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.172212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:24.172357 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:24.172380 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 09:18:24.172488 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 09:18:24.172614 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":3,"thread_start_us":575,"threads_started":1}
I20260925 09:18:24.174444 20841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0252ed18d0bd4b4e984ef16d6bb93f61" candidate_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e133b1e4215b436c81f53688f1646649" is_pre_election: true
I20260925 09:18:24.174436 20708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0252ed18d0bd4b4e984ef16d6bb93f61" candidate_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb" is_pre_election: true
I20260925 09:18:24.174891 20841 raft_consensus.cc:2454] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46031b71cdcc4e92bfc1947a8ecd7d0e in term 0.
I20260925 09:18:24.174899 20708 raft_consensus.cc:2454] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46031b71cdcc4e92bfc1947a8ecd7d0e in term 0.
I20260925 09:18:24.175511 20510 leader_election.cc:302] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46031b71cdcc4e92bfc1947a8ecd7d0e, c0ba8d1b46f243e0b7457bf76ae99bcb; no voters:
I20260925 09:18:24.175930 20913 raft_consensus.cc:2781] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:18:24.176080 20913 raft_consensus.cc:482] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:18:24.176204 20913 raft_consensus.cc:3037] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:24.178645 20913 raft_consensus.cc:504] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33153 } } peers { permanent_uuid: "e133b1e4215b436c81f53688f1646649" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41895 } } peers { permanent_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 41321 } }
I20260925 09:18:24.179327 20913 leader_election.cc:288] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [CANDIDATE]: Term 1 election: Requested vote from peers c0ba8d1b46f243e0b7457bf76ae99bcb (127.3.19.66:33153), e133b1e4215b436c81f53688f1646649 (127.3.19.67:41895)
I20260925 09:18:24.179734 20708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0252ed18d0bd4b4e984ef16d6bb93f61" candidate_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb"
I20260925 09:18:24.179966 20708 raft_consensus.cc:3037] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:24.179970 20841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0252ed18d0bd4b4e984ef16d6bb93f61" candidate_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e133b1e4215b436c81f53688f1646649"
I20260925 09:18:24.180202 20841 raft_consensus.cc:3037] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:24.182395 20708 raft_consensus.cc:2454] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46031b71cdcc4e92bfc1947a8ecd7d0e in term 1.
I20260925 09:18:24.182508 20841 raft_consensus.cc:2454] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46031b71cdcc4e92bfc1947a8ecd7d0e in term 1.
I20260925 09:18:24.182857 20510 leader_election.cc:302] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46031b71cdcc4e92bfc1947a8ecd7d0e, c0ba8d1b46f243e0b7457bf76ae99bcb; no voters:
I20260925 09:18:24.183152 20913 raft_consensus.cc:2781] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:24.183558 20913 raft_consensus.cc:686] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [term 1 LEADER]: Becoming Leader. State: Replica: 46031b71cdcc4e92bfc1947a8ecd7d0e, State: Running, Role: LEADER
I20260925 09:18:24.184021 20913 consensus_queue.cc:237] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33153 } } peers { permanent_uuid: "e133b1e4215b436c81f53688f1646649" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41895 } } peers { permanent_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 41321 } }
I20260925 09:18:24.188830 20429 catalog_manager.cc:5495] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 46031b71cdcc4e92bfc1947a8ecd7d0e (127.3.19.65). New cstate: current_term: 1 leader_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e133b1e4215b436c81f53688f1646649" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46031b71cdcc4e92bfc1947a8ecd7d0e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 41321 } health_report { overall_health: HEALTHY } } }
I20260925 09:18:24.258038 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:24.252516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.3.19.65:41321)
0925 09:18:24.252780 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:24.252787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:24.252883 (+ 96us) server_negotiation.cc:407] Connection header received
0925 09:18:24.253130 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:24.253134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:24.253212 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:24.253383 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:18:24.253765 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.254491 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.255873 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.256286 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.256421 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:24.256929 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:18:24.257459 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:18:24.257641 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:24.257691 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 09:18:24.257770 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 09:18:24.266062 20708 raft_consensus.cc:1261] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb [term 1 FOLLOWER]: Refusing update from remote peer 46031b71cdcc4e92bfc1947a8ecd7d0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:18:24.266062 20841 raft_consensus.cc:1261] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649 [term 1 FOLLOWER]: Refusing update from remote peer 46031b71cdcc4e92bfc1947a8ecd7d0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:18:24.267177 20916 consensus_queue.cc:1035] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e133b1e4215b436c81f53688f1646649" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:18:24.267571 20913 consensus_queue.cc:1035] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ba8d1b46f243e0b7457bf76ae99bcb" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:18:24.292013 20430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:18:24.292309 20430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:18:24.295765 20430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 241066d46f1e45aba9055930bd614e93: Sending DeleteTablet for 3 replicas of tablet 0252ed18d0bd4b4e984ef16d6bb93f61
I20260925 09:18:24.297197 20688 tablet_service.cc:1514] Processing DeleteTablet for tablet 0252ed18d0bd4b4e984ef16d6bb93f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:24 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:53034
I20260925 09:18:24.297206 20821 tablet_service.cc:1514] Processing DeleteTablet for tablet 0252ed18d0bd4b4e984ef16d6bb93f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:24 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:54444
I20260925 09:18:24.297254 20554 tablet_service.cc:1514] Processing DeleteTablet for tablet 0252ed18d0bd4b4e984ef16d6bb93f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:24 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:36674
I20260925 09:18:24.299594 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 20491
I20260925 09:18:24.308113 20929 tablet_replica.cc:330] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb: stopping tablet replica
I20260925 09:18:24.308660 20929 raft_consensus.cc:2229] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:18:24.309221 20929 raft_consensus.cc:2258] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:24.310144 20930 tablet_replica.cc:330] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649: stopping tablet replica
I20260925 09:18:24.310627 20930 raft_consensus.cc:2229] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:18:24.311167 20930 raft_consensus.cc:2258] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:24.313838 20929 ts_tablet_manager.cc:1905] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:24.315667 20930 ts_tablet_manager.cc:1905] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:24.320660 20929 ts_tablet_manager.cc:1918] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:24.320812 20929 log.cc:1196] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/0252ed18d0bd4b4e984ef16d6bb93f61
I20260925 09:18:24.321373 20929 ts_tablet_manager.cc:1939] T 0252ed18d0bd4b4e984ef16d6bb93f61 P c0ba8d1b46f243e0b7457bf76ae99bcb: Deleting consensus metadata
I20260925 09:18:24.323190 20417 catalog_manager.cc:4841] TS c0ba8d1b46f243e0b7457bf76ae99bcb (127.3.19.66:33153): tablet 0252ed18d0bd4b4e984ef16d6bb93f61 (table test-table [id=f3c513e531324f63be5920d46b1112e2]) successfully deleted
I20260925 09:18:24.324087 20931 tablet_replica.cc:330] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e: stopping tablet replica
I20260925 09:18:24.327481 20931 raft_consensus.cc:2229] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:18:24.328298 20931 raft_consensus.cc:2258] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:24.328765 20930 ts_tablet_manager.cc:1918] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:24.328944 20930 log.cc:1196] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/0252ed18d0bd4b4e984ef16d6bb93f61
I20260925 09:18:24.329499 20930 ts_tablet_manager.cc:1939] T 0252ed18d0bd4b4e984ef16d6bb93f61 P e133b1e4215b436c81f53688f1646649: Deleting consensus metadata
I20260925 09:18:24.330755 20931 ts_tablet_manager.cc:1905] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:24.331158 20415 catalog_manager.cc:4841] TS e133b1e4215b436c81f53688f1646649 (127.3.19.67:41895): tablet 0252ed18d0bd4b4e984ef16d6bb93f61 (table test-table [id=f3c513e531324f63be5920d46b1112e2]) successfully deleted
Sep 25 09:18:24 dist-test-slave-l6qh krb5kdc[20385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:18:24.337849 20931 ts_tablet_manager.cc:1918] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:24.338028 20931 log.cc:1196] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/0252ed18d0bd4b4e984ef16d6bb93f61
I20260925 09:18:24.338581 20931 ts_tablet_manager.cc:1939] T 0252ed18d0bd4b4e984ef16d6bb93f61 P 46031b71cdcc4e92bfc1947a8ecd7d0e: Deleting consensus metadata
I20260925 09:18:24.340240 20415 catalog_manager.cc:4841] TS 46031b71cdcc4e92bfc1947a8ecd7d0e (127.3.19.65:41321): tablet 0252ed18d0bd4b4e984ef16d6bb93f61 (table test-table [id=f3c513e531324f63be5920d46b1112e2]) successfully deleted
I20260925 09:18:24.342779 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:24.301329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.3.19.65:41321)
0925 09:18:24.301549 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:24.301559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:24.316785 (+ 15226us) server_negotiation.cc:407] Connection header received
0925 09:18:24.320922 (+ 4137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:24.320929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:24.321023 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:24.321114 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:24.323336 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.324693 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.326721 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.327451 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.327593 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:24.335184 (+ 7591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:24.335196 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:24.335218 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:24.335267 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:24.339680 (+ 4413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:24.340539 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:24.340545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:24.340547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:24.340698 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:24.341110 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:24.341113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:24.341127 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:24.341526 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:24.341751 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:24.342193 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 09:18:24.342427 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 09:18:24.343525 20614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:24.440053 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 20491
I20260925 09:18:24.458845 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 20625
Sep 25 09:18:24 dist-test-slave-l6qh krb5kdc[20385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:18:24.472371 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:24.459587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.3.19.66:33153)
0925 09:18:24.459765 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:24.459770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:24.459786 (+ 16us) server_negotiation.cc:407] Connection header received
0925 09:18:24.459911 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:24.459915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:24.459978 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:24.460066 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:24.461293 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.461983 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.463216 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.463584 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.463687 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:24.467796 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:24.467807 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:24.467821 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:24.467849 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:24.470270 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:24.470793 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:24.470798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:24.470801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:24.470911 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:24.471197 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:24.471199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:24.471201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:24.471528 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:24.471765 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:24.471965 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 09:18:24.472109 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 09:18:24.472994 20748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:24.561144 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 20625
I20260925 09:18:24.578392 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 20758
Sep 25 09:18:24 dist-test-slave-l6qh krb5kdc[20385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:18:24.592499 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:24.579397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.3.19.67:41895)
0925 09:18:24.579593 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:24.579597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:24.579615 (+ 18us) server_negotiation.cc:407] Connection header received
0925 09:18:24.579757 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:24.579761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:24.579834 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:24.579918 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:24.581205 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.581941 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.583121 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:24.583478 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:24.583564 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:24.587740 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:24.587754 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:24.587770 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:24.587798 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:24.590271 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:24.590870 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:24.590878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:24.590880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:24.591019 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:24.591313 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:24.591316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:24.591317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:24.591623 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:24.591819 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:24.592066 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 09:18:24.592236 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 09:18:24.593083 20881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:24.677703 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 20758
I20260925 09:18:24.695128 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 20398
I20260925 09:18:24.696269 20460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:24.803543 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 20398
2026-09-25T09:18:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4603 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T09:18:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:24Z Disabled control of system clock
I20260925 09:18:24.880638 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:42851
--webserver_interface=127.3.19.126
--webserver_port=0
--builtin_ntp_servers=127.3.19.84:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:42851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 09:18:25.200508 20940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:25.200785 20940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:25.200947 20940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:25.208464 20940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:25.208550 20940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:25.208586 20940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:25.208670 20940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:25.208731 20940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:25.217257 20940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:42851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:42851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:25.218483 20940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:25.220240 20940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:25.231310 20946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:25.231415 20945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:25.231410 20948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:25.232254 20940 server_base.cc:1034] running on GCE node
I20260925 09:18:25.232913 20940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:25.234243 20940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:25.235466 20940 hybrid_clock.cc:648] HybridClock initialized: now 1790327905235432 us; error 49 us; skew 500 ppm
I20260925 09:18:25.235811 20940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:18:25.241185 20940 webserver.cc:460] Webserver started at http://127.3.19.126:42263/ using document root <none> and password file <none>
I20260925 09:18:25.241885 20940 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:25.242010 20940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:25.242336 20940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:25.245100 20940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "ea69f359a106469aa98e954624eaa6a0"
format_stamp: "Formatted at 2026-09-25 09:18:25 on dist-test-slave-l6qh"
server_key: "742412d964c38ab1273ff3bf98ab1267"
server_key_iv: "657c1524f9047ecd5ae0ac325d5d4c04"
server_key_version: "encryptionkey@0"
I20260925 09:18:25.245949 20940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "ea69f359a106469aa98e954624eaa6a0"
format_stamp: "Formatted at 2026-09-25 09:18:25 on dist-test-slave-l6qh"
server_key: "742412d964c38ab1273ff3bf98ab1267"
server_key_iv: "657c1524f9047ecd5ae0ac325d5d4c04"
server_key_version: "encryptionkey@0"
I20260925 09:18:25.250618 20940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 09:18:25.253916 20954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:25.255199 20940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:18:25.255381 20940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "ea69f359a106469aa98e954624eaa6a0"
format_stamp: "Formatted at 2026-09-25 09:18:25 on dist-test-slave-l6qh"
server_key: "742412d964c38ab1273ff3bf98ab1267"
server_key_iv: "657c1524f9047ecd5ae0ac325d5d4c04"
server_key_version: "encryptionkey@0"
I20260925 09:18:25.255589 20940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:25.268316 20940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:25.269047 20940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:25.269435 20940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:25.287518 20940 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:42851
I20260925 09:18:25.287549 21005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:42851 every 8 connection(s)
I20260925 09:18:25.288751 20940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:25.291762 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 20940
I20260925 09:18:25.291929 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:25.292527 3149 external_mini_cluster.cc:1428] Setting key 5e0e38f34ee9a09b0d15d995b281384d
I20260925 09:18:25.292760 21006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:25.305871 21006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0: Bootstrap starting.
I20260925 09:18:25.306767 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:25.294388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.3.19.126:42851)
0925 09:18:25.295337 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:25.295347 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:25.295389 (+ 42us) server_negotiation.cc:407] Connection header received
0925 09:18:25.297186 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:25.297210 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:25.297680 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:25.297996 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:25.298576 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:25.300382 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:25.302279 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:25.303259 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:25.303449 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:25.304625 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:25.304632 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:25.304644 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:25.305125 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:25.305197 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:25.305217 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:25.305380 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:25.305604 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 09:18:25.305888 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":358,"threads_started":1}
I20260925 09:18:25.308828 21006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:25.309894 21006 log.cc:824] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:25.312184 21006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0: No bootstrap required, opened a new log
I20260925 09:18:25.319262 21006 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea69f359a106469aa98e954624eaa6a0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42851 } }
I20260925 09:18:25.319628 21006 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:25.319734 21006 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea69f359a106469aa98e954624eaa6a0, State: Initialized, Role: FOLLOWER
I20260925 09:18:25.320236 21006 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea69f359a106469aa98e954624eaa6a0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42851 } }
I20260925 09:18:25.320483 21006 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:25.320621 21006 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:25.320811 21006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:25.322949 21006 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea69f359a106469aa98e954624eaa6a0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42851 } }
I20260925 09:18:25.323390 21006 leader_election.cc:302] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea69f359a106469aa98e954624eaa6a0; no voters:
I20260925 09:18:25.323918 21006 leader_election.cc:288] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:25.324067 21011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:25.324781 21011 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [term 1 LEADER]: Becoming Leader. State: Replica: ea69f359a106469aa98e954624eaa6a0, State: Running, Role: LEADER
I20260925 09:18:25.325328 21011 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea69f359a106469aa98e954624eaa6a0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42851 } }
I20260925 09:18:25.325947 21006 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:25.327993 21012 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea69f359a106469aa98e954624eaa6a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea69f359a106469aa98e954624eaa6a0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42851 } } }
I20260925 09:18:25.328475 21012 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:25.328377 21013 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea69f359a106469aa98e954624eaa6a0. Latest consensus state: current_term: 1 leader_uuid: "ea69f359a106469aa98e954624eaa6a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea69f359a106469aa98e954624eaa6a0" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42851 } } }
I20260925 09:18:25.328679 21013 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:25.331280 21018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:25.336072 21018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:25.344152 21018 catalog_manager.cc:1347] Generated new cluster ID: 34fc840b6a3b4542975ddecfb559c6eb
I20260925 09:18:25.344269 21018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:25.369741 21018 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:25.370946 21018 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:25.377478 21018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0: Generated new TSK 0
I20260925 09:18:25.378487 21018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:18:25.390100 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:42851
--builtin_ntp_servers=127.3.19.84:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 09:18:25.726428 21030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:25.726805 21030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:25.727029 21030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:25.734390 21030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:25.734494 21030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:25.734670 21030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:18:25.743562 21030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:42851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:25.744762 21030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:25.746549 21030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:25.759704 21035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:25.759639 21036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:25.759734 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:25.760607 21030 server_base.cc:1034] running on GCE node
I20260925 09:18:25.761345 21030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:25.762306 21030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:25.763543 21030 hybrid_clock.cc:648] HybridClock initialized: now 1790327905763495 us; error 59 us; skew 500 ppm
I20260925 09:18:25.763893 21030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:18:25.766187 21030 webserver.cc:460] Webserver started at http://127.3.19.65:44157/ using document root <none> and password file <none>
I20260925 09:18:25.766882 21030 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:25.767014 21030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:25.767333 21030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:25.770212 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "1b6c2fc51459469f9f21d5e87fed282a"
format_stamp: "Formatted at 2026-09-25 09:18:25 on dist-test-slave-l6qh"
server_key: "64d82ffa3cdc15f60acd3fa507311bde"
server_key_iv: "afae6447b8994eecb635787b9e6b4741"
server_key_version: "encryptionkey@0"
I20260925 09:18:25.771258 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "1b6c2fc51459469f9f21d5e87fed282a"
format_stamp: "Formatted at 2026-09-25 09:18:25 on dist-test-slave-l6qh"
server_key: "64d82ffa3cdc15f60acd3fa507311bde"
server_key_iv: "afae6447b8994eecb635787b9e6b4741"
server_key_version: "encryptionkey@0"
I20260925 09:18:25.776150 21030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 09:18:25.779492 21044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:25.781167 21030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:18:25.781364 21030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "1b6c2fc51459469f9f21d5e87fed282a"
format_stamp: "Formatted at 2026-09-25 09:18:25 on dist-test-slave-l6qh"
server_key: "64d82ffa3cdc15f60acd3fa507311bde"
server_key_iv: "afae6447b8994eecb635787b9e6b4741"
server_key_version: "encryptionkey@0"
I20260925 09:18:25.781608 21030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:25.795499 21030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:25.796254 21030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:25.796520 21030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:25.797473 21030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:25.798841 21030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:25.798957 21030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:25.799088 21030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:25.799175 21030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:25.832254 21030 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:36939
I20260925 09:18:25.832360 21156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:36939 every 8 connection(s)
I20260925 09:18:25.833611 21030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:18:25.836998 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 21030
I20260925 09:18:25.837185 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:18:25.837782 3149 external_mini_cluster.cc:1428] Setting key 4ef205d016f63fdc20e7158f2d1b31f4
I20260925 09:18:25.840890 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:42851
--builtin_ntp_servers=127.3.19.84:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 09:18:25.848065 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:25.836083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:58887 (local address 127.3.19.126:42851)
0925 09:18:25.836531 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:25.836538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:25.838160 (+ 1622us) server_negotiation.cc:407] Connection header received
0925 09:18:25.840399 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:25.840403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:25.840463 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:25.840544 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:25.841957 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:25.842609 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:25.845047 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:25.845517 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:25.845606 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:25.847178 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:25.847186 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:25.847192 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:25.847213 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:25.847257 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:25.847272 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:25.847346 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:25.847588 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 09:18:25.847679 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":219,"threads_started":1}
I20260925 09:18:25.849452 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:25.836788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:42851 (local address 127.3.19.65:58887)
0925 09:18:25.837989 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:25.838036 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:25.840101 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:25.840836 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:25.840848 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:25.840965 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:25.841793 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:25.841818 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:25.842774 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:25.842782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:25.844909 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:25.844925 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:25.845727 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:25.845736 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:25.846059 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:25.846819 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:18:25.846824 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:25.846874 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:18:25.846984 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:25.847372 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:25.847386 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:25.847396 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 09:18:25.847568 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 09:18:25.847978 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":376,"threads_started":1}
I20260925 09:18:25.850883 21157 heartbeater.cc:347] Connected to a master server at 127.3.19.126:42851
I20260925 09:18:25.851163 21157 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:25.851852 21157 heartbeater.cc:511] Master 127.3.19.126:42851 requested a full tablet report, sending...
I20260925 09:18:25.854060 20971 ts_manager.cc:194] Registered new tserver with Master: 1b6c2fc51459469f9f21d5e87fed282a (127.3.19.65:36939)
I20260925 09:18:25.856287 20971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.19.65:58887
W20260925 09:18:26.166880 21161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:26.167133 21161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:26.167223 21161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:26.174172 21161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:26.174255 21161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:26.174357 21161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:18:26.182989 21161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:42851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:26.184106 21161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:26.186000 21161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:26.198714 21169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:26.198771 21167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:26.198741 21166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:26.199265 21161 server_base.cc:1034] running on GCE node
I20260925 09:18:26.199909 21161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:26.200790 21161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:26.202014 21161 hybrid_clock.cc:648] HybridClock initialized: now 1790327906201972 us; error 52 us; skew 500 ppm
I20260925 09:18:26.202340 21161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:18:26.204540 21161 webserver.cc:460] Webserver started at http://127.3.19.66:39081/ using document root <none> and password file <none>
I20260925 09:18:26.205189 21161 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:26.205380 21161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:26.205704 21161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:26.208348 21161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "5391b1db32f8480b9b66d631a93b66f9"
format_stamp: "Formatted at 2026-09-25 09:18:26 on dist-test-slave-l6qh"
server_key: "fe4ffeae71b212fc2bfdd82ae680900d"
server_key_iv: "7cb9af458e0b73c7b29595f026f2073b"
server_key_version: "encryptionkey@0"
I20260925 09:18:26.209088 21161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "5391b1db32f8480b9b66d631a93b66f9"
format_stamp: "Formatted at 2026-09-25 09:18:26 on dist-test-slave-l6qh"
server_key: "fe4ffeae71b212fc2bfdd82ae680900d"
server_key_iv: "7cb9af458e0b73c7b29595f026f2073b"
server_key_version: "encryptionkey@0"
I20260925 09:18:26.213951 21161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 09:18:26.217161 21175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:26.218444 21161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 09:18:26.218621 21161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "5391b1db32f8480b9b66d631a93b66f9"
format_stamp: "Formatted at 2026-09-25 09:18:26 on dist-test-slave-l6qh"
server_key: "fe4ffeae71b212fc2bfdd82ae680900d"
server_key_iv: "7cb9af458e0b73c7b29595f026f2073b"
server_key_version: "encryptionkey@0"
I20260925 09:18:26.218842 21161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:26.231302 21161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:26.232108 21161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:26.232414 21161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:26.233377 21161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:26.234728 21161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:26.234848 21161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:26.234972 21161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:26.235066 21161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:26.267637 21161 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:43963
I20260925 09:18:26.267684 21287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:43963 every 8 connection(s)
I20260925 09:18:26.268990 21161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:18:26.269389 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 21161
I20260925 09:18:26.273926 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:18:26.274402 3149 external_mini_cluster.cc:1428] Setting key d465d4845b9838d601d7f200ccaaba27
I20260925 09:18:26.277379 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:42851
--builtin_ntp_servers=127.3.19.84:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 09:18:26.283108 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.271781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:44219 (local address 127.3.19.126:42851)
0925 09:18:26.271995 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:26.272001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:26.273642 (+ 1641us) server_negotiation.cc:407] Connection header received
0925 09:18:26.275685 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:26.275689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:26.275749 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:26.275837 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:26.277044 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.278224 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.279917 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.280590 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.280739 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.282157 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:26.282163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:26.282172 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:26.282196 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:26.282237 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:26.282255 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:26.282364 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:26.282553 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 09:18:26.282697 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":70}
I20260925 09:18:26.283867 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.272364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:42851 (local address 127.3.19.66:44219)
0925 09:18:26.273462 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:26.273505 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:26.275392 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:26.276091 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:26.276104 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:26.276267 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:26.276884 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.276900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.278357 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.278360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.279776 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.279787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.280786 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.280791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.280981 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.281792 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:18:26.281795 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:26.281832 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:18:26.281903 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:26.282362 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:26.282369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:26.282375 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 09:18:26.282511 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 09:18:26.282811 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":371,"threads_started":1}
I20260925 09:18:26.285313 21288 heartbeater.cc:347] Connected to a master server at 127.3.19.126:42851
I20260925 09:18:26.285605 21288 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:26.286362 21288 heartbeater.cc:511] Master 127.3.19.126:42851 requested a full tablet report, sending...
I20260925 09:18:26.287936 20971 ts_manager.cc:194] Registered new tserver with Master: 5391b1db32f8480b9b66d631a93b66f9 (127.3.19.66:43963)
I20260925 09:18:26.289036 20971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.19.66:44219
W20260925 09:18:26.597510 21291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:26.597805 21291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:26.597963 21291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:26.605005 21291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:26.605131 21291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:26.605302 21291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:18:26.613914 21291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:42851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:26.615163 21291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:26.616882 21291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:26.629817 21296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:26.629887 21297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:26.629855 21299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:26.631072 21291 server_base.cc:1034] running on GCE node
I20260925 09:18:26.631656 21291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:26.632678 21291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:26.633939 21291 hybrid_clock.cc:648] HybridClock initialized: now 1790327906633855 us; error 87 us; skew 500 ppm
I20260925 09:18:26.634366 21291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:18:26.637048 21291 webserver.cc:460] Webserver started at http://127.3.19.67:40885/ using document root <none> and password file <none>
I20260925 09:18:26.637919 21291 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:26.638042 21291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:26.638412 21291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:26.641920 21291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "896dffcebe554f1ea206ec1530b3d065"
format_stamp: "Formatted at 2026-09-25 09:18:26 on dist-test-slave-l6qh"
server_key: "f7566b0fdcb0d2dfd2ec4b0c8f3674ed"
server_key_iv: "628631809401dbaf00bfadb6160845d2"
server_key_version: "encryptionkey@0"
I20260925 09:18:26.642664 21291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "896dffcebe554f1ea206ec1530b3d065"
format_stamp: "Formatted at 2026-09-25 09:18:26 on dist-test-slave-l6qh"
server_key: "f7566b0fdcb0d2dfd2ec4b0c8f3674ed"
server_key_iv: "628631809401dbaf00bfadb6160845d2"
server_key_version: "encryptionkey@0"
I20260925 09:18:26.647466 21291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 09:18:26.650906 21305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:26.652287 21291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:18:26.652415 21291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "896dffcebe554f1ea206ec1530b3d065"
format_stamp: "Formatted at 2026-09-25 09:18:26 on dist-test-slave-l6qh"
server_key: "f7566b0fdcb0d2dfd2ec4b0c8f3674ed"
server_key_iv: "628631809401dbaf00bfadb6160845d2"
server_key_version: "encryptionkey@0"
I20260925 09:18:26.652581 21291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:26.665395 21291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:26.666078 21291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:26.666296 21291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:26.667090 21291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:26.668367 21291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:26.668447 21291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:26.668520 21291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:26.668571 21291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:26.700877 21291 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:36689
I20260925 09:18:26.700944 21417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:36689 every 8 connection(s)
I20260925 09:18:26.702203 21291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:18:26.703104 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 21291
I20260925 09:18:26.703284 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:18:26.703895 3149 external_mini_cluster.cc:1428] Setting key dd7c4125f69af8f5f8c66126a51c5ec7
I20260925 09:18:26.714565 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.705037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:37309 (local address 127.3.19.126:42851)
0925 09:18:26.705302 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:26.705308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:26.706967 (+ 1659us) server_negotiation.cc:407] Connection header received
0925 09:18:26.708680 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:26.708684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:26.708747 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:26.708826 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:26.709840 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.710591 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.712059 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.712489 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.712588 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.713869 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:26.713873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:26.713877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:26.713893 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:26.713926 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:26.713938 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:26.714007 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:26.714211 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 09:18:26.714285 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260925 09:18:26.715406 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.705650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:42851 (local address 127.3.19.67:37309)
0925 09:18:26.706813 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:26.706860 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:26.708451 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:26.709014 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:26.709021 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:26.709101 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:26.709708 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.709723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.710715 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.710718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.711911 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.711922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.712669 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.712674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.712863 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.713597 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:18:26.713600 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:26.713638 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:18:26.713705 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:26.714020 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:26.714027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:26.714032 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 09:18:26.714170 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 09:18:26.714435 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":324,"threads_started":1}
I20260925 09:18:26.716701 21418 heartbeater.cc:347] Connected to a master server at 127.3.19.126:42851
I20260925 09:18:26.716969 21418 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:26.717782 21418 heartbeater.cc:511] Master 127.3.19.126:42851 requested a full tablet report, sending...
I20260925 09:18:26.719256 20970 ts_manager.cc:194] Registered new tserver with Master: 896dffcebe554f1ea206ec1530b3d065 (127.3.19.67:36689)
I20260925 09:18:26.720151 20970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.19.67:37309
I20260925 09:18:26.731206 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:18:26.740538 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.735189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.3.19.126:42851)
0925 09:18:26.735343 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:26.735348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:26.735802 (+ 454us) server_negotiation.cc:407] Connection header received
0925 09:18:26.735932 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:26.735936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:26.736003 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:26.736078 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:26.736501 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.737171 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.738504 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.738914 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.739008 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.740015 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:26.740019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:26.740023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:26.740038 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:26.740067 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:26.740078 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:26.740154 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:26.740269 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 09:18:26.740333 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 09:18:26.744423 20970 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:18:26.751537 20970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:18:26.787624 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.766026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.3.19.66:43963)
0925 09:18:26.778857 (+ 12831us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:26.778865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:26.779764 (+ 899us) server_negotiation.cc:407] Connection header received
0925 09:18:26.780001 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:26.780010 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:26.780242 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:26.780434 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:26.781317 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.782584 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.785494 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.786518 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.786664 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.786900 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:26.787127 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 09:18:26.787280 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12675,"thread_start_us":12573,"threads_started":1}
I20260925 09:18:26.787652 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.765319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:43963 (local address 127.0.0.1:50054)
0925 09:18:26.779648 (+ 14329us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:26.779662 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:26.779802 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:26.780746 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:26.780750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:26.780774 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:26.781138 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.781149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.782721 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.782732 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.785126 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.785168 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.786696 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.786701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.786879 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.786925 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 09:18:26.787062 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 09:18:26.787245 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14210,"mutex_wait_us":61,"thread_start_us":6227,"threads_started":1}
I20260925 09:18:26.788261 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.763620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:36689 (local address 127.0.0.1:37546)
0925 09:18:26.777217 (+ 13597us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:26.777255 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:26.777370 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:26.779514 (+ 2144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:26.779525 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:26.779563 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:26.780306 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.780322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.782290 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.782294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.785362 (+ 3068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.785374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.787364 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.787368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.787660 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.787695 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 09:18:26.787788 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 09:18:26.787913 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13418,"mutex_wait_us":277,"thread_start_us":13861,"threads_started":1}
I20260925 09:18:26.789019 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.764240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.3.19.67:36689)
0925 09:18:26.778560 (+ 14320us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:26.778571 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:26.778609 (+ 38us) server_negotiation.cc:407] Connection header received
0925 09:18:26.778704 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:26.778710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:26.778979 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:26.779193 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:26.780525 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.782152 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.785532 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.786836 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.786963 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.787226 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:26.788501 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0925 09:18:26.788663 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14144,"thread_start_us":13989,"threads_started":1}
I20260925 09:18:26.789248 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.763337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:36939 (local address 127.0.0.1:46182)
0925 09:18:26.771947 (+ 8610us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:26.771987 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:26.772186 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:26.779853 (+ 7667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:26.779860 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:26.779887 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:26.780358 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.780377 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.782117 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.782123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.785102 (+ 2979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.785117 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.787355 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.787361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.788649 (+ 1288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.788695 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 09:18:26.788783 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 09:18:26.788906 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8347,"mutex_wait_us":1245,"thread_start_us":16052,"threads_started":1}
I20260925 09:18:26.789354 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.763640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.3.19.65:36939)
0925 09:18:26.779238 (+ 15598us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:26.779249 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:26.779278 (+ 29us) server_negotiation.cc:407] Connection header received
0925 09:18:26.779400 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:26.779407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:26.779630 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:26.779873 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:26.780587 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.781987 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.785321 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.786836 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.786963 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.787182 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:26.788822 (+ 1640us) server_negotiation.cc:299] Negotiation successful
0925 09:18:26.788988 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15447,"thread_start_us":15347,"threads_started":1}
I20260925 09:18:26.791531 21223 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf1a7fda0854be4a9c800a01c9d0ec6 (DEFAULT_TABLE table=test-table [id=2551f913994b4d86b2f8acef9649f529]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:26.791810 21353 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf1a7fda0854be4a9c800a01c9d0ec6 (DEFAULT_TABLE table=test-table [id=2551f913994b4d86b2f8acef9649f529]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:26.792753 21223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf1a7fda0854be4a9c800a01c9d0ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:26.793048 21353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf1a7fda0854be4a9c800a01c9d0ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:26.797025 21092 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf1a7fda0854be4a9c800a01c9d0ec6 (DEFAULT_TABLE table=test-table [id=2551f913994b4d86b2f8acef9649f529]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:26.798341 21092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf1a7fda0854be4a9c800a01c9d0ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:26.803486 21436 tablet_bootstrap.cc:492] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065: Bootstrap starting.
I20260925 09:18:26.806874 21436 tablet_bootstrap.cc:654] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:26.807865 21436 log.cc:824] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:26.810484 21436 tablet_bootstrap.cc:492] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065: No bootstrap required, opened a new log
I20260925 09:18:26.810724 21436 ts_tablet_manager.cc:1397] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 09:18:26.812093 21439 tablet_bootstrap.cc:492] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a: Bootstrap starting.
I20260925 09:18:26.814363 21437 tablet_bootstrap.cc:492] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9: Bootstrap starting.
I20260925 09:18:26.815083 21439 tablet_bootstrap.cc:654] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:26.816095 21439 log.cc:824] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:26.817719 21437 tablet_bootstrap.cc:654] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:26.818841 21439 tablet_bootstrap.cc:492] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a: No bootstrap required, opened a new log
I20260925 09:18:26.818823 21437 log.cc:824] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:26.819097 21439 ts_tablet_manager.cc:1397] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260925 09:18:26.819550 21436 raft_consensus.cc:348] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6c2fc51459469f9f21d5e87fed282a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36939 } } peers { permanent_uuid: "896dffcebe554f1ea206ec1530b3d065" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36689 } } peers { permanent_uuid: "5391b1db32f8480b9b66d631a93b66f9" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43963 } }
I20260925 09:18:26.819905 21436 raft_consensus.cc:374] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:26.820036 21436 raft_consensus.cc:729] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896dffcebe554f1ea206ec1530b3d065, State: Initialized, Role: FOLLOWER
I20260925 09:18:26.820681 21436 consensus_queue.cc:260] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6c2fc51459469f9f21d5e87fed282a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36939 } } peers { permanent_uuid: "896dffcebe554f1ea206ec1530b3d065" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36689 } } peers { permanent_uuid: "5391b1db32f8480b9b66d631a93b66f9" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43963 } }
I20260925 09:18:26.821795 21437 tablet_bootstrap.cc:492] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9: No bootstrap required, opened a new log
I20260925 09:18:26.821769 21418 heartbeater.cc:503] Master 127.3.19.126:42851 was elected leader, sending a full tablet report...
I20260925 09:18:26.822077 21437 ts_tablet_manager.cc:1397] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260925 09:18:26.822520 21436 ts_tablet_manager.cc:1428] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065: Time spent starting tablet: real 0.012s user 0.006s sys 0.007s
I20260925 09:18:26.829624 21439 raft_consensus.cc:348] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6c2fc51459469f9f21d5e87fed282a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36939 } } peers { permanent_uuid: "896dffcebe554f1ea206ec1530b3d065" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36689 } } peers { permanent_uuid: "5391b1db32f8480b9b66d631a93b66f9" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43963 } }
I20260925 09:18:26.829962 21439 raft_consensus.cc:374] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:26.830061 21439 raft_consensus.cc:729] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6c2fc51459469f9f21d5e87fed282a, State: Initialized, Role: FOLLOWER
I20260925 09:18:26.830421 21437 raft_consensus.cc:348] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6c2fc51459469f9f21d5e87fed282a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36939 } } peers { permanent_uuid: "896dffcebe554f1ea206ec1530b3d065" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36689 } } peers { permanent_uuid: "5391b1db32f8480b9b66d631a93b66f9" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43963 } }
I20260925 09:18:26.830732 21437 raft_consensus.cc:374] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:26.830610 21439 consensus_queue.cc:260] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6c2fc51459469f9f21d5e87fed282a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36939 } } peers { permanent_uuid: "896dffcebe554f1ea206ec1530b3d065" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36689 } } peers { permanent_uuid: "5391b1db32f8480b9b66d631a93b66f9" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43963 } }
I20260925 09:18:26.830842 21437 raft_consensus.cc:729] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5391b1db32f8480b9b66d631a93b66f9, State: Initialized, Role: FOLLOWER
I20260925 09:18:26.831461 21437 consensus_queue.cc:260] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6c2fc51459469f9f21d5e87fed282a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36939 } } peers { permanent_uuid: "896dffcebe554f1ea206ec1530b3d065" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36689 } } peers { permanent_uuid: "5391b1db32f8480b9b66d631a93b66f9" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43963 } }
I20260925 09:18:26.831959 21157 heartbeater.cc:503] Master 127.3.19.126:42851 was elected leader, sending a full tablet report...
I20260925 09:18:26.832839 21439 ts_tablet_manager.cc:1428] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 09:18:26.833822 21288 heartbeater.cc:503] Master 127.3.19.126:42851 was elected leader, sending a full tablet report...
I20260925 09:18:26.834240 21437 ts_tablet_manager.cc:1428] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260925 09:18:26.835783 21158 tablet.cc:2369] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:18:26.841672 21444 raft_consensus.cc:482] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:18:26.841918 21444 raft_consensus.cc:504] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6c2fc51459469f9f21d5e87fed282a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36939 } } peers { permanent_uuid: "896dffcebe554f1ea206ec1530b3d065" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36689 } } peers { permanent_uuid: "5391b1db32f8480b9b66d631a93b66f9" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43963 } }
I20260925 09:18:26.843478 21444 leader_election.cc:288] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b6c2fc51459469f9f21d5e87fed282a (127.3.19.65:36939), 896dffcebe554f1ea206ec1530b3d065 (127.3.19.67:36689)
I20260925 09:18:26.849539 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.844206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:34141 (local address 127.3.19.67:36689)
0925 09:18:26.844387 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:26.844395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:26.844452 (+ 57us) server_negotiation.cc:407] Connection header received
0925 09:18:26.844721 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:26.844727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:26.844798 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:26.844893 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:26.845469 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.846187 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.847945 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.848684 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.848813 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.848913 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:26.849191 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 09:18:26.849327 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 09:18:26.849632 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.843749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:36689 (local address 127.3.19.66:34141)
0925 09:18:26.844357 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:26.844377 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:26.844563 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:26.844907 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:26.844912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:26.844935 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:26.845332 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.845342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.846326 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.846333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.847775 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.847786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.848872 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.848878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.849027 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.849049 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 09:18:26.849170 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 09:18:26.849310 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":2,"thread_start_us":650,"threads_started":1}
I20260925 09:18:26.849704 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.843960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:57919 (local address 127.3.19.65:36939)
0925 09:18:26.844187 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:26.844197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:26.844802 (+ 605us) server_negotiation.cc:407] Connection header received
0925 09:18:26.844982 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:26.844986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:26.845060 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:26.845161 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:26.845789 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.846448 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.848126 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.848830 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.848958 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.849049 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:26.849354 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 09:18:26.849479 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 09:18:26.849786 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.843785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:36939 (local address 127.3.19.66:57919)
0925 09:18:26.844702 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:26.844720 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:26.844855 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:26.845177 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:26.845181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:26.845206 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:26.845647 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.845655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.846602 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.846607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.847951 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:26.847962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.849015 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.849022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:26.849180 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.849207 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 09:18:26.849321 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 09:18:26.849434 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":22,"thread_start_us":612,"threads_started":1}
I20260925 09:18:26.850637 21373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf1a7fda0854be4a9c800a01c9d0ec6" candidate_uuid: "5391b1db32f8480b9b66d631a93b66f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896dffcebe554f1ea206ec1530b3d065" is_pre_election: true
I20260925 09:18:26.850816 21112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf1a7fda0854be4a9c800a01c9d0ec6" candidate_uuid: "5391b1db32f8480b9b66d631a93b66f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b6c2fc51459469f9f21d5e87fed282a" is_pre_election: true
I20260925 09:18:26.851202 21373 raft_consensus.cc:2454] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5391b1db32f8480b9b66d631a93b66f9 in term 0.
I20260925 09:18:26.851298 21112 raft_consensus.cc:2454] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5391b1db32f8480b9b66d631a93b66f9 in term 0.
I20260925 09:18:26.851907 21178 leader_election.cc:302] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b6c2fc51459469f9f21d5e87fed282a, 5391b1db32f8480b9b66d631a93b66f9; no voters:
I20260925 09:18:26.852293 21444 raft_consensus.cc:2781] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:18:26.852413 21444 raft_consensus.cc:482] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:18:26.852483 21444 raft_consensus.cc:3037] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:26.854851 21444 raft_consensus.cc:504] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6c2fc51459469f9f21d5e87fed282a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36939 } } peers { permanent_uuid: "896dffcebe554f1ea206ec1530b3d065" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36689 } } peers { permanent_uuid: "5391b1db32f8480b9b66d631a93b66f9" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43963 } }
I20260925 09:18:26.855403 21444 leader_election.cc:288] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [CANDIDATE]: Term 1 election: Requested vote from peers 1b6c2fc51459469f9f21d5e87fed282a (127.3.19.65:36939), 896dffcebe554f1ea206ec1530b3d065 (127.3.19.67:36689)
I20260925 09:18:26.855803 21112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf1a7fda0854be4a9c800a01c9d0ec6" candidate_uuid: "5391b1db32f8480b9b66d631a93b66f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b6c2fc51459469f9f21d5e87fed282a"
I20260925 09:18:26.855885 21373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf1a7fda0854be4a9c800a01c9d0ec6" candidate_uuid: "5391b1db32f8480b9b66d631a93b66f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896dffcebe554f1ea206ec1530b3d065"
I20260925 09:18:26.855979 21112 raft_consensus.cc:3037] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:26.856120 21373 raft_consensus.cc:3037] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:26.858383 21112 raft_consensus.cc:2454] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5391b1db32f8480b9b66d631a93b66f9 in term 1.
I20260925 09:18:26.858417 21373 raft_consensus.cc:2454] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5391b1db32f8480b9b66d631a93b66f9 in term 1.
I20260925 09:18:26.858754 21178 leader_election.cc:302] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b6c2fc51459469f9f21d5e87fed282a, 5391b1db32f8480b9b66d631a93b66f9; no voters:
I20260925 09:18:26.859061 21444 raft_consensus.cc:2781] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:26.859549 21444 raft_consensus.cc:686] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [term 1 LEADER]: Becoming Leader. State: Replica: 5391b1db32f8480b9b66d631a93b66f9, State: Running, Role: LEADER
I20260925 09:18:26.859962 21444 consensus_queue.cc:237] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6c2fc51459469f9f21d5e87fed282a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36939 } } peers { permanent_uuid: "896dffcebe554f1ea206ec1530b3d065" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36689 } } peers { permanent_uuid: "5391b1db32f8480b9b66d631a93b66f9" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43963 } }
I20260925 09:18:26.864315 20971 catalog_manager.cc:5495] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5391b1db32f8480b9b66d631a93b66f9 (127.3.19.66). New cstate: current_term: 1 leader_uuid: "5391b1db32f8480b9b66d631a93b66f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6c2fc51459469f9f21d5e87fed282a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "896dffcebe554f1ea206ec1530b3d065" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5391b1db32f8480b9b66d631a93b66f9" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 43963 } health_report { overall_health: HEALTHY } } }
I20260925 09:18:26.884680 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.879241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.3.19.66:43963)
0925 09:18:26.879425 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:26.879430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:26.879446 (+ 16us) server_negotiation.cc:407] Connection header received
0925 09:18:26.879626 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:26.879629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:26.879712 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:26.879873 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:18:26.880215 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.880909 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.882532 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.882965 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.883066 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.883575 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:18:26.884062 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:18:26.884177 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:26.884244 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 09:18:26.884342 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 09:18:26.892454 21112 raft_consensus.cc:1261] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a [term 1 FOLLOWER]: Refusing update from remote peer 5391b1db32f8480b9b66d631a93b66f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:18:26.892454 21373 raft_consensus.cc:1261] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065 [term 1 FOLLOWER]: Refusing update from remote peer 5391b1db32f8480b9b66d631a93b66f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:18:26.893424 21444 consensus_queue.cc:1035] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b6c2fc51459469f9f21d5e87fed282a" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:18:26.893711 21447 consensus_queue.cc:1035] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "896dffcebe554f1ea206ec1530b3d065" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 36689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:18:26.916307 20970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:18:26.916585 20970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:18:26.919920 20970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea69f359a106469aa98e954624eaa6a0: Sending DeleteTablet for 3 replicas of tablet 6bf1a7fda0854be4a9c800a01c9d0ec6
I20260925 09:18:26.920964 21353 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf1a7fda0854be4a9c800a01c9d0ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:26 UTC) from {username='slave'} at 127.0.0.1:37546
I20260925 09:18:26.920951 21092 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf1a7fda0854be4a9c800a01c9d0ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:26 UTC) from {username='slave'} at 127.0.0.1:46182
I20260925 09:18:26.921938 21223 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf1a7fda0854be4a9c800a01c9d0ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:26 UTC) from {username='slave'} at 127.0.0.1:50054
I20260925 09:18:26.922299 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 21030
I20260925 09:18:26.933795 21460 tablet_replica.cc:330] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a: stopping tablet replica
I20260925 09:18:26.934304 21460 raft_consensus.cc:2229] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:18:26.934892 21460 raft_consensus.cc:2258] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:26.935792 21459 tablet_replica.cc:330] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065: stopping tablet replica
I20260925 09:18:26.936199 21461 tablet_replica.cc:330] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9: stopping tablet replica
I20260925 09:18:26.936208 21459 raft_consensus.cc:2229] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:18:26.936679 21459 raft_consensus.cc:2258] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:26.937904 21460 ts_tablet_manager.cc:1905] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:26.938640 21461 raft_consensus.cc:2229] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:18:26.938688 21459 ts_tablet_manager.cc:1905] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:26.939374 21461 raft_consensus.cc:2258] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:26.942986 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:26.923837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.3.19.65:36939)
0925 09:18:26.924066 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:26.924075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:26.934551 (+ 10476us) server_negotiation.cc:407] Connection header received
0925 09:18:26.935048 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:26.935057 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:26.935184 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:26.935274 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:26.935914 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.937030 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.938859 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:26.939568 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:26.939670 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:26.941193 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:26.941203 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:26.941218 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:26.942063 (+ 845us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:26.942116 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:26.942171 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:26.942253 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:26.942492 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 09:18:26.942595 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 09:18:26.943671 21461 ts_tablet_manager.cc:1905] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:26.944119 21152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:26.953116 21459 ts_tablet_manager.cc:1918] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:26.953392 21459 log.cc:1196] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/6bf1a7fda0854be4a9c800a01c9d0ec6
I20260925 09:18:26.953619 21461 ts_tablet_manager.cc:1918] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:26.953837 21461 log.cc:1196] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/6bf1a7fda0854be4a9c800a01c9d0ec6
I20260925 09:18:26.954015 21459 ts_tablet_manager.cc:1939] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 896dffcebe554f1ea206ec1530b3d065: Deleting consensus metadata
I20260925 09:18:26.954396 21461 ts_tablet_manager.cc:1939] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 5391b1db32f8480b9b66d631a93b66f9: Deleting consensus metadata
I20260925 09:18:26.955920 20956 catalog_manager.cc:4841] TS 896dffcebe554f1ea206ec1530b3d065 (127.3.19.67:36689): tablet 6bf1a7fda0854be4a9c800a01c9d0ec6 (table test-table [id=2551f913994b4d86b2f8acef9649f529]) successfully deleted
I20260925 09:18:26.956027 20958 catalog_manager.cc:4841] TS 5391b1db32f8480b9b66d631a93b66f9 (127.3.19.66:43963): tablet 6bf1a7fda0854be4a9c800a01c9d0ec6 (table test-table [id=2551f913994b4d86b2f8acef9649f529]) successfully deleted
I20260925 09:18:27.059377 21460 ts_tablet_manager.cc:1918] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:27.059651 21460 log.cc:1196] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/6bf1a7fda0854be4a9c800a01c9d0ec6
I20260925 09:18:27.060268 21460 ts_tablet_manager.cc:1939] T 6bf1a7fda0854be4a9c800a01c9d0ec6 P 1b6c2fc51459469f9f21d5e87fed282a: Deleting consensus metadata
I20260925 09:18:27.061401 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 21030
W20260925 09:18:27.084774 20957 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.19.65:36939 (error 108)
I20260925 09:18:27.085556 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 21161
I20260925 09:18:27.086828 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:27.086247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:36939 (local address 127.0.0.1:46192)
0925 09:18:27.086517 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:27.086661 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260925 09:18:27.087064 20957 catalog_manager.cc:4584] TS 1b6c2fc51459469f9f21d5e87fed282a (127.3.19.65:36939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6bf1a7fda0854be4a9c800a01c9d0ec6: Network error: Client connection negotiation failed: client connection to 127.3.19.65:36939: connect: Connection refused (error 111)
I20260925 09:18:27.092760 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:27.086384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.3.19.66:43963)
0925 09:18:27.086610 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:27.086618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:27.086637 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:18:27.086857 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:27.086861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:27.086981 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:27.087083 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:27.087553 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:27.088339 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:27.089787 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:27.090199 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:27.090330 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:27.091359 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:27.091367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:27.091384 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:27.092036 (+ 652us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:27.092080 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:27.092098 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:27.092220 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:27.092358 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 09:18:27.092455 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260925 09:18:27.093492 21283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:27.140733 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:27.140202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:36939 (local address 127.0.0.1:46200)
0925 09:18:27.140423 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:27.140545 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 09:18:27.141971 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:27.141593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:36939 (local address 127.0.0.1:46202)
0925 09:18:27.141762 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:27.141846 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 09:18:27.201287 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 21161
I20260925 09:18:27.202494 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:27.201985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:36939 (local address 127.0.0.1:46208)
0925 09:18:27.202226 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:27.202330 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 09:18:27.203408 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:27.203074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:36939 (local address 127.0.0.1:46210)
0925 09:18:27.203232 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:27.203306 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 09:18:27.220512 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 21291
I20260925 09:18:27.226935 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:27.221332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37548 (local address 127.3.19.67:36689)
0925 09:18:27.221503 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:27.221508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:27.221525 (+ 17us) server_negotiation.cc:407] Connection header received
0925 09:18:27.221691 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:27.221694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:27.221772 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:27.221856 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:27.222206 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:27.222904 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:27.224119 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:27.224510 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:27.224603 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:27.225712 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:27.225720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:27.225735 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:27.226271 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:27.226320 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:27.226339 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:27.226424 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:27.226535 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 09:18:27.226621 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 09:18:27.227514 21413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:27.283636 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:27.283116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:36939 (local address 127.0.0.1:46224)
0925 09:18:27.283334 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:27.283417 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 09:18:27.284785 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:27.284388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:36939 (local address 127.0.0.1:46236)
0925 09:18:27.284612 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:27.284681 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 09:18:27.319418 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 21291
I20260925 09:18:27.336735 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 20940
I20260925 09:18:27.337837 21001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:27.450981 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 20940
2026-09-25T09:18:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2645 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:18:27 dist-test-slave-l6qh krb5kdc[21470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:18:27 dist-test-slave-l6qh krb5kdc[21470](info): set up 2 sockets
Sep 25 09:18:27 dist-test-slave-l6qh krb5kdc[21470](info): commencing operation
krb5kdc: starting...
W20260925 09:18:28.558526 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:18:28 dist-test-slave-l6qh krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:18:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:18:28.803990 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33395
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:33395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:29.126787 21483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:29.127137 21483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:29.127301 21483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:29.134389 21483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:29.134541 21483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:29.134653 21483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:29.134763 21483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:29.134852 21483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:29.144542 21483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:33395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:29.145896 21483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:29.147185 21483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:29.158871 21488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:29.158782 21491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:29.158825 21489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:29.159885 21483 server_base.cc:1034] running on GCE node
I20260925 09:18:29.160629 21483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:29.162024 21483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:29.163249 21483 hybrid_clock.cc:648] HybridClock initialized: now 1790327909163213 us; error 46 us; skew 500 ppm
I20260925 09:18:29.163571 21483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:29 dist-test-slave-l6qh krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327909, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:29.166693 21483 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:18:29.168020 21483 webserver.cc:460] Webserver started at http://127.3.19.126:42789/ using document root <none> and password file <none>
I20260925 09:18:29.168673 21483 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:29.168804 21483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:29.169111 21483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:29.171913 21483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "8984ccd51d844f86af3be163fc574ff8"
format_stamp: "Formatted at 2026-09-25 09:18:29 on dist-test-slave-l6qh"
server_key: "35a6185c714779ea54c1b157ca231ea9"
server_key_iv: "edbf2d7c39db13c36cc0e14a7ac5c151"
server_key_version: "encryptionkey@0"
I20260925 09:18:29.172638 21483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "8984ccd51d844f86af3be163fc574ff8"
format_stamp: "Formatted at 2026-09-25 09:18:29 on dist-test-slave-l6qh"
server_key: "35a6185c714779ea54c1b157ca231ea9"
server_key_iv: "edbf2d7c39db13c36cc0e14a7ac5c151"
server_key_version: "encryptionkey@0"
I20260925 09:18:29.177595 21483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 09:18:29.180779 21498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:29.182086 21483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:18:29.182276 21483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "8984ccd51d844f86af3be163fc574ff8"
format_stamp: "Formatted at 2026-09-25 09:18:29 on dist-test-slave-l6qh"
server_key: "35a6185c714779ea54c1b157ca231ea9"
server_key_iv: "edbf2d7c39db13c36cc0e14a7ac5c151"
server_key_version: "encryptionkey@0"
I20260925 09:18:29.182468 21483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:29.207006 21483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:29.211249 21483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:29.211517 21483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:29.233803 21483 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:33395
I20260925 09:18:29.233835 21549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:33395 every 8 connection(s)
I20260925 09:18:29.235141 21483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:29.237498 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 21483
I20260925 09:18:29.237694 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:29.238173 3149 external_mini_cluster.cc:1428] Setting key 1f8c32765b6d53c07eeb9b7de0093483
I20260925 09:18:29.239145 21550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 09:18:29 dist-test-slave-l6qh krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:18:29.252866 21550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8: Bootstrap starting.
I20260925 09:18:29.256884 21550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:29.258217 21550 log.cc:824] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:29.261531 21550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8: No bootstrap required, opened a new log
I20260925 09:18:29.262496 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:29.239979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.3.19.126:33395)
0925 09:18:29.240769 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:29.240785 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:29.240833 (+ 48us) server_negotiation.cc:407] Connection header received
0925 09:18:29.242683 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:29.242710 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:29.243210 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:29.243647 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:29.244953 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:29.247059 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:29.248349 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:29.249396 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:29.249597 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:29.253985 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:29.253995 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:29.254011 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:29.254051 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:29.257964 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:29.258756 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:29.258768 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:29.258780 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:29.258966 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:29.259448 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:29.259451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:29.259454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:29.260142 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:29.260426 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:29.260832 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 09:18:29.261252 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":326,"threads_started":1}
I20260925 09:18:29.269568 21550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8984ccd51d844f86af3be163fc574ff8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33395 } }
I20260925 09:18:29.269906 21550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:29.269979 21550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8984ccd51d844f86af3be163fc574ff8, State: Initialized, Role: FOLLOWER
I20260925 09:18:29.270465 21550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8984ccd51d844f86af3be163fc574ff8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33395 } }
I20260925 09:18:29.270711 21550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:29.270854 21550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:29.271049 21550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:29.273360 21550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8984ccd51d844f86af3be163fc574ff8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33395 } }
I20260925 09:18:29.273842 21550 leader_election.cc:302] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8984ccd51d844f86af3be163fc574ff8; no voters:
I20260925 09:18:29.274415 21550 leader_election.cc:288] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:29.274546 21555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:29.275823 21555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [term 1 LEADER]: Becoming Leader. State: Replica: 8984ccd51d844f86af3be163fc574ff8, State: Running, Role: LEADER
I20260925 09:18:29.276412 21555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8984ccd51d844f86af3be163fc574ff8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33395 } }
I20260925 09:18:29.276623 21550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:29.279345 21556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8984ccd51d844f86af3be163fc574ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8984ccd51d844f86af3be163fc574ff8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33395 } } }
I20260925 09:18:29.279742 21556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:29.279421 21557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8984ccd51d844f86af3be163fc574ff8. Latest consensus state: current_term: 1 leader_uuid: "8984ccd51d844f86af3be163fc574ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8984ccd51d844f86af3be163fc574ff8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33395 } } }
I20260925 09:18:29.279870 21557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:29.281733 21562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:29.288322 21562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:29.297575 21562 catalog_manager.cc:1347] Generated new cluster ID: 70d0761043844fd0b68a6cce862bf924
I20260925 09:18:29.297700 21562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:29.318192 21562 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:29.320039 21562 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:29.329336 21562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8: Generated new TSK 0
I20260925 09:18:29.330353 21562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:18:29.388532 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33395
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:29.717775 21576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:29.718029 21576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:29.718201 21576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:29.725173 21576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:29.725279 21576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:29.725459 21576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:18:29.734156 21576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:29.735306 21576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:29.736572 21576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:29.749039 21582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:29.749104 21581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:29.749643 21584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:29.749920 21576 server_base.cc:1034] running on GCE node
I20260925 09:18:29.750495 21576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:29.751287 21576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:29.752482 21576 hybrid_clock.cc:648] HybridClock initialized: now 1790327909752424 us; error 47 us; skew 500 ppm
I20260925 09:18:29.752817 21576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:29 dist-test-slave-l6qh krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327909, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:29.755810 21576 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:18:29.757030 21576 webserver.cc:460] Webserver started at http://127.3.19.65:41267/ using document root <none> and password file <none>
I20260925 09:18:29.757679 21576 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:29.757792 21576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:29.758033 21576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:29.760697 21576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "003a69dc1f3b4de0961c0ef55d49d770"
format_stamp: "Formatted at 2026-09-25 09:18:29 on dist-test-slave-l6qh"
server_key: "1b0bb987eb20b57d08d4383fb1d6f74b"
server_key_iv: "70d14a31d32e3eaff738864fa5e2a6e1"
server_key_version: "encryptionkey@0"
I20260925 09:18:29.761564 21576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "003a69dc1f3b4de0961c0ef55d49d770"
format_stamp: "Formatted at 2026-09-25 09:18:29 on dist-test-slave-l6qh"
server_key: "1b0bb987eb20b57d08d4383fb1d6f74b"
server_key_iv: "70d14a31d32e3eaff738864fa5e2a6e1"
server_key_version: "encryptionkey@0"
I20260925 09:18:29.766489 21576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 09:18:29.769626 21591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:29.770797 21576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260925 09:18:29.770954 21576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "003a69dc1f3b4de0961c0ef55d49d770"
format_stamp: "Formatted at 2026-09-25 09:18:29 on dist-test-slave-l6qh"
server_key: "1b0bb987eb20b57d08d4383fb1d6f74b"
server_key_iv: "70d14a31d32e3eaff738864fa5e2a6e1"
server_key_version: "encryptionkey@0"
I20260925 09:18:29.771127 21576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:29.789330 21576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:29.793620 21576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:29.793908 21576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:29.794776 21576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:29.796042 21576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:29.796157 21576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:29.796300 21576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:29.796367 21576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:29.829610 21576 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:33531
I20260925 09:18:29.829710 21703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:33531 every 8 connection(s)
I20260925 09:18:29.831009 21576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:18:29.834070 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 21576
I20260925 09:18:29.834314 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:18:29.834913 3149 external_mini_cluster.cc:1428] Setting key 312193adc10a9f5722fe12159bfcdd61
Sep 25 09:18:29 dist-test-slave-l6qh krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327909, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:18:29.851722 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:29.833515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:36563 (local address 127.3.19.126:33395)
0925 09:18:29.834014 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:29.834019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:29.835587 (+ 1568us) server_negotiation.cc:407] Connection header received
0925 09:18:29.837652 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:29.837659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:29.837745 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:29.837888 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:29.839717 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:29.840461 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:29.842091 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:29.842489 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:29.842686 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:29.846245 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:29.846249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:29.846253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:29.846285 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:29.848796 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:29.849545 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:29.849551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:29.849558 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:29.849672 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:29.850083 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:29.850088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:29.850091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:29.850337 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:29.850504 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:29.851160 (+ 656us) server_negotiation.cc:299] Negotiation successful
0925 09:18:29.851370 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":268,"threads_started":1}
I20260925 09:18:29.852573 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:29.834021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33395 (local address 127.3.19.65:36563)
0925 09:18:29.835367 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:29.835455 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:29.837343 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:29.838147 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:29.838155 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:29.838932 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:29.839516 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:29.839535 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:29.840634 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:29.840639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:29.841942 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:29.841955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:29.842814 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:29.842824 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:29.843078 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:29.843771 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:18:29.843797 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:29.846025 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:29.848929 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:29.848937 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:29.848951 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:29.849428 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:29.849781 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:29.849786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:29.849790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:29.849970 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:29.850459 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:29.850485 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:29.850808 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 09:18:29.851053 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 09:18:29.851536 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":503,"threads_started":1}
I20260925 09:18:29.854251 21704 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33395
I20260925 09:18:29.854532 21704 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:29.855249 21704 heartbeater.cc:511] Master 127.3.19.126:33395 requested a full tablet report, sending...
I20260925 09:18:29.857314 21515 ts_manager.cc:194] Registered new tserver with Master: 003a69dc1f3b4de0961c0ef55d49d770 (127.3.19.65:33531)
I20260925 09:18:29.858633 21515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:36563
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:18:29.891801 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33395
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:30.211053 21710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:30.211311 21710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:30.211400 21710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:30.218266 21710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:30.218357 21710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:30.218458 21710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:18:30.227470 21710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:30.228554 21710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:30.229809 21710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:30.243142 21716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:30.243402 21715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:30.243724 21718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:30.244249 21710 server_base.cc:1034] running on GCE node
I20260925 09:18:30.244807 21710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:30.245759 21710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:30.246981 21710 hybrid_clock.cc:648] HybridClock initialized: now 1790327910246937 us; error 38 us; skew 500 ppm
I20260925 09:18:30.247292 21710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:30 dist-test-slave-l6qh krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327910, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:30.257882 21710 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:18:30.259230 21710 webserver.cc:460] Webserver started at http://127.3.19.66:46865/ using document root <none> and password file <none>
I20260925 09:18:30.259871 21710 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:30.259965 21710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:30.260221 21710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:30.262965 21710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "48659d6002764413b1e1aaeed0a99e6a"
format_stamp: "Formatted at 2026-09-25 09:18:30 on dist-test-slave-l6qh"
server_key: "49e19529f461ff3a24a1353676741a73"
server_key_iv: "2ab925e4c0ea29997f8c04bdd136e733"
server_key_version: "encryptionkey@0"
I20260925 09:18:30.263665 21710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "48659d6002764413b1e1aaeed0a99e6a"
format_stamp: "Formatted at 2026-09-25 09:18:30 on dist-test-slave-l6qh"
server_key: "49e19529f461ff3a24a1353676741a73"
server_key_iv: "2ab925e4c0ea29997f8c04bdd136e733"
server_key_version: "encryptionkey@0"
I20260925 09:18:30.268702 21710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:18:30.272070 21725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:30.273348 21710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:18:30.273499 21710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "48659d6002764413b1e1aaeed0a99e6a"
format_stamp: "Formatted at 2026-09-25 09:18:30 on dist-test-slave-l6qh"
server_key: "49e19529f461ff3a24a1353676741a73"
server_key_iv: "2ab925e4c0ea29997f8c04bdd136e733"
server_key_version: "encryptionkey@0"
I20260925 09:18:30.273680 21710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:30.291522 21710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:30.295743 21710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:30.296033 21710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:30.296907 21710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:30.298213 21710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:30.298286 21710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:30.298458 21710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:30.298507 21710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:30.330983 21710 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:42521
I20260925 09:18:30.331067 21837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:42521 every 8 connection(s)
I20260925 09:18:30.332324 21710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:18:30.335505 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 21710
I20260925 09:18:30.335726 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:18:30.336398 3149 external_mini_cluster.cc:1428] Setting key 63cbbf03de4bd5100e8b1f1c5c5e3059
Sep 25 09:18:30 dist-test-slave-l6qh krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327910, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:18:30.353606 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:30.335178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:38475 (local address 127.3.19.126:33395)
0925 09:18:30.335375 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:30.335380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:30.336907 (+ 1527us) server_negotiation.cc:407] Connection header received
0925 09:18:30.339228 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:30.339233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:30.339330 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:30.339436 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:30.340870 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.341643 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.343224 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.343633 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.343782 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:30.347318 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:30.347322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:30.347325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:30.347368 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:30.350170 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:30.351011 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:30.351014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:30.351017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:30.351156 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:30.351632 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:30.351635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:30.351637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:30.351886 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:30.352000 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:30.352879 (+ 879us) server_negotiation.cc:299] Negotiation successful
0925 09:18:30.353165 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 09:18:30.355053 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:30.335931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33395 (local address 127.3.19.66:38475)
0925 09:18:30.336750 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:30.336792 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:30.338964 (+ 2172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:30.339674 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:30.339683 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:30.340167 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:30.340705 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:30.340723 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.341804 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.341808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:30.343062 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:30.343073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.343900 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.343908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:30.344144 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:30.344891 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:18:30.344912 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:30.347068 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:30.350301 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:30.350311 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:30.350328 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:30.350900 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:30.351273 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:30.351280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:30.351286 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:30.351518 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:30.352019 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:30.352029 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:30.352465 (+ 436us) client_negotiation.cc:769] Sending connection context
0925 09:18:30.352770 (+ 305us) client_negotiation.cc:240] Negotiation successful
0925 09:18:30.353705 (+ 935us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":319,"threads_started":1}
I20260925 09:18:30.356567 21838 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33395
I20260925 09:18:30.356899 21838 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:30.357810 21838 heartbeater.cc:511] Master 127.3.19.126:33395 requested a full tablet report, sending...
I20260925 09:18:30.359375 21515 ts_manager.cc:194] Registered new tserver with Master: 48659d6002764413b1e1aaeed0a99e6a (127.3.19.66:42521)
I20260925 09:18:30.360327 21515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:38475
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:18:30.396171 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33395
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:30.712795 21843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:30.713096 21843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:30.713196 21843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:30.720896 21843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:30.721015 21843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:30.721181 21843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:18:30.730307 21843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:30.731455 21843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:30.732645 21843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:30.745306 21849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:30.745333 21851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:30.745563 21848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:30.746138 21843 server_base.cc:1034] running on GCE node
I20260925 09:18:30.746680 21843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:30.747577 21843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:30.748799 21843 hybrid_clock.cc:648] HybridClock initialized: now 1790327910748766 us; error 44 us; skew 500 ppm
I20260925 09:18:30.749091 21843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:30 dist-test-slave-l6qh krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327910, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:30.752038 21843 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:18:30.753473 21843 webserver.cc:460] Webserver started at http://127.3.19.67:45559/ using document root <none> and password file <none>
I20260925 09:18:30.754124 21843 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:30.754298 21843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:30.754668 21843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:30.757479 21843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "4afb74ae9198452db84ada9cb748a394"
format_stamp: "Formatted at 2026-09-25 09:18:30 on dist-test-slave-l6qh"
server_key: "3077914f49591e098cce318a74cb6651"
server_key_iv: "559f50597a5c6c00d93c73d395a354e5"
server_key_version: "encryptionkey@0"
I20260925 09:18:30.758195 21843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "4afb74ae9198452db84ada9cb748a394"
format_stamp: "Formatted at 2026-09-25 09:18:30 on dist-test-slave-l6qh"
server_key: "3077914f49591e098cce318a74cb6651"
server_key_iv: "559f50597a5c6c00d93c73d395a354e5"
server_key_version: "encryptionkey@0"
I20260925 09:18:30.762818 21843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 09:18:30.765908 21858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:30.767166 21843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:18:30.767333 21843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "4afb74ae9198452db84ada9cb748a394"
format_stamp: "Formatted at 2026-09-25 09:18:30 on dist-test-slave-l6qh"
server_key: "3077914f49591e098cce318a74cb6651"
server_key_iv: "559f50597a5c6c00d93c73d395a354e5"
server_key_version: "encryptionkey@0"
I20260925 09:18:30.767515 21843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:30.788337 21843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:30.792234 21843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:30.792492 21843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:30.793412 21843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:30.794541 21843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:30.794639 21843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:30.794791 21843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:30.794881 21843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:30.827741 21843 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:41577
I20260925 09:18:30.827791 21970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:41577 every 8 connection(s)
I20260925 09:18:30.829015 21843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:18:30.829487 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 21843
I20260925 09:18:30.829711 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:18:30.830404 3149 external_mini_cluster.cc:1428] Setting key 1a5dbb6563733423a6e41ba05ee14c7b
Sep 25 09:18:30 dist-test-slave-l6qh krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327910, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:18:30.851214 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:30.832870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:58725 (local address 127.3.19.126:33395)
0925 09:18:30.833046 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:30.833053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:30.837064 (+ 4011us) server_negotiation.cc:407] Connection header received
0925 09:18:30.838832 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:30.838835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:30.838922 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:30.839004 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:30.840381 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.841103 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.842637 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.843018 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.843109 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:30.846408 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:30.846412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:30.846416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:30.846445 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:30.848612 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:30.849331 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:30.849337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:30.849342 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:30.849478 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:30.849849 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:30.849852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:30.849854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:30.850122 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:30.850231 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:30.850811 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 09:18:30.850979 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 09:18:30.852068 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:30.832311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33395 (local address 127.3.19.67:58725)
0925 09:18:30.836903 (+ 4592us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:30.836946 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:30.838605 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:30.839210 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:30.839219 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:30.839705 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:30.840234 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:30.840250 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.841315 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.841319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:30.842519 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:30.842530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.843198 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.843203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:30.843406 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:30.844071 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:18:30.844091 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:30.846224 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:30.848723 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:30.848729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:30.848739 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:30.849116 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:30.849574 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:30.849577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:30.849579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:30.849749 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:30.850220 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:30.850225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:30.850493 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 09:18:30.850707 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 09:18:30.851048 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4291,"thread_start_us":4064,"threads_started":1}
I20260925 09:18:30.853438 21971 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33395
I20260925 09:18:30.853686 21971 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:30.854282 21971 heartbeater.cc:511] Master 127.3.19.126:33395 requested a full tablet report, sending...
I20260925 09:18:30.855784 21515 ts_manager.cc:194] Registered new tserver with Master: 4afb74ae9198452db84ada9cb748a394 (127.3.19.67:41577)
I20260925 09:18:30.856766 21515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:58725
I20260925 09:18:30.858443 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:18:30.861505 21704 heartbeater.cc:503] Master 127.3.19.126:33395 was elected leader, sending a full tablet report...
I20260925 09:18:30.874830 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:30.862472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.3.19.126:33395)
0925 09:18:30.862637 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:30.862643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:30.863028 (+ 385us) server_negotiation.cc:407] Connection header received
0925 09:18:30.863200 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:30.863205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:30.863267 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:30.863362 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:30.864685 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.865481 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.866781 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.867168 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.867297 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:30.869769 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:30.869780 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:30.869783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:30.869811 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:30.872181 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:30.872856 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:30.872863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:30.872871 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:30.872979 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:30.873494 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:30.873502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:30.873504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:30.873774 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:30.874023 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:30.874363 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 09:18:30.874518 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 09:18:30.878597 21514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:18:30.885672 21514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:18:30.925689 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:30.897009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:41577 (local address 127.0.0.1:51310)
0925 09:18:30.910286 (+ 13277us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:30.910334 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:30.910542 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:30.916115 (+ 5573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:30.916126 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:30.916174 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:30.916792 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:30.916809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.919240 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.919248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:30.922098 (+ 2850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:30.922117 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.923932 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.923938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:30.924726 (+ 788us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:30.924796 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 09:18:30.924936 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 09:18:30.925203 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12960,"thread_start_us":12789,"threads_started":1}
I20260925 09:18:30.925845 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:30.897323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51310 (local address 127.3.19.67:41577)
0925 09:18:30.915307 (+ 17984us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:30.915320 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:30.915366 (+ 46us) server_negotiation.cc:407] Connection header received
0925 09:18:30.915488 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:30.915499 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:30.915842 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:30.916086 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:30.917047 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.919049 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.922336 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.923758 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.923888 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:30.924108 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:30.925350 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0925 09:18:30.925520 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17757,"thread_start_us":17567,"threads_started":1}
I20260925 09:18:30.929518 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:30.897213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:42521 (local address 127.0.0.1:53962)
0925 09:18:30.920521 (+ 23308us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:30.920540 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:30.920660 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:30.922837 (+ 2177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:30.922842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:30.922865 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:30.923273 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:30.923288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.925190 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.925195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:30.927186 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:30.927198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.928709 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.928716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:30.928917 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:30.928946 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 09:18:30.929031 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 09:18:30.929145 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23171,"mutex_wait_us":25,"thread_start_us":26012,"threads_started":1}
I20260925 09:18:30.930230 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:30.897465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.3.19.66:42521)
0925 09:18:30.922131 (+ 24666us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:30.922143 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:30.922178 (+ 35us) server_negotiation.cc:407] Connection header received
0925 09:18:30.922292 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:30.922303 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:30.922602 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:30.922838 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:30.923512 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.925031 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.927344 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.928211 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.928369 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:30.928559 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:30.929904 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0925 09:18:30.930029 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24462,"thread_start_us":24341,"threads_started":1}
I20260925 09:18:30.930266 21906 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4c27e7cf554905b6b9d22280513cc3 (DEFAULT_TABLE table=test-table [id=de80731c6b8b4adc9beb173740e5be37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:30.931533 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:30.899385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:33531 (local address 127.0.0.1:59574)
0925 09:18:30.923540 (+ 24155us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:30.923553 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:30.923693 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:30.924256 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:30.924261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:30.924284 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:30.924675 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:30.924686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.926791 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.926795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:30.928848 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:30.928861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.930720 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.930726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:30.930900 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:30.930943 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 09:18:30.931037 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 09:18:30.931160 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24017,"mutex_wait_us":1,"thread_start_us":20868,"threads_started":1}
I20260925 09:18:30.931576 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:30.899599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.3.19.65:33531)
0925 09:18:30.922444 (+ 22845us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:30.922452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:30.923660 (+ 1208us) server_negotiation.cc:407] Connection header received
0925 09:18:30.923844 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:30.923853 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:30.924059 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:30.924255 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:30.924882 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.926656 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.929029 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:30.930515 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:30.930647 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:30.930866 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:30.931088 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 09:18:30.931236 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22663,"thread_start_us":22838,"threads_started":1}
I20260925 09:18:30.931845 21906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4c27e7cf554905b6b9d22280513cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:30.933487 21773 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4c27e7cf554905b6b9d22280513cc3 (DEFAULT_TABLE table=test-table [id=de80731c6b8b4adc9beb173740e5be37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:30.934789 21773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4c27e7cf554905b6b9d22280513cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:30.935050 21639 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4c27e7cf554905b6b9d22280513cc3 (DEFAULT_TABLE table=test-table [id=de80731c6b8b4adc9beb173740e5be37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:30.936337 21639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4c27e7cf554905b6b9d22280513cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:30.943156 21989 tablet_bootstrap.cc:492] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394: Bootstrap starting.
I20260925 09:18:30.946619 21989 tablet_bootstrap.cc:654] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:30.947664 21989 log.cc:824] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:30.950320 21989 tablet_bootstrap.cc:492] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394: No bootstrap required, opened a new log
I20260925 09:18:30.950491 21991 tablet_bootstrap.cc:492] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a: Bootstrap starting.
I20260925 09:18:30.950604 21989 ts_tablet_manager.cc:1397] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260925 09:18:30.951103 21992 tablet_bootstrap.cc:492] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770: Bootstrap starting.
I20260925 09:18:30.953860 21991 tablet_bootstrap.cc:654] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:30.953974 21992 tablet_bootstrap.cc:654] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:30.955034 21991 log.cc:824] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:30.955262 21992 log.cc:824] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:30.958055 21991 tablet_bootstrap.cc:492] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a: No bootstrap required, opened a new log
I20260925 09:18:30.958312 21992 tablet_bootstrap.cc:492] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770: No bootstrap required, opened a new log
I20260925 09:18:30.958312 21991 ts_tablet_manager.cc:1397] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 09:18:30.958552 21992 ts_tablet_manager.cc:1397] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 09:18:30.960021 21989 raft_consensus.cc:348] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4afb74ae9198452db84ada9cb748a394" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41577 } } peers { permanent_uuid: "48659d6002764413b1e1aaeed0a99e6a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 42521 } } peers { permanent_uuid: "003a69dc1f3b4de0961c0ef55d49d770" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33531 } }
I20260925 09:18:30.960424 21989 raft_consensus.cc:374] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:30.960536 21989 raft_consensus.cc:729] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4afb74ae9198452db84ada9cb748a394, State: Initialized, Role: FOLLOWER
I20260925 09:18:30.961277 21989 consensus_queue.cc:260] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4afb74ae9198452db84ada9cb748a394" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41577 } } peers { permanent_uuid: "48659d6002764413b1e1aaeed0a99e6a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 42521 } } peers { permanent_uuid: "003a69dc1f3b4de0961c0ef55d49d770" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33531 } }
I20260925 09:18:30.963544 21971 heartbeater.cc:503] Master 127.3.19.126:33395 was elected leader, sending a full tablet report...
I20260925 09:18:30.964046 21989 ts_tablet_manager.cc:1428] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260925 09:18:30.967690 21992 raft_consensus.cc:348] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4afb74ae9198452db84ada9cb748a394" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41577 } } peers { permanent_uuid: "48659d6002764413b1e1aaeed0a99e6a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 42521 } } peers { permanent_uuid: "003a69dc1f3b4de0961c0ef55d49d770" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33531 } }
I20260925 09:18:30.968101 21992 raft_consensus.cc:374] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:30.968216 21992 raft_consensus.cc:729] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003a69dc1f3b4de0961c0ef55d49d770, State: Initialized, Role: FOLLOWER
I20260925 09:18:30.968792 21992 consensus_queue.cc:260] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4afb74ae9198452db84ada9cb748a394" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41577 } } peers { permanent_uuid: "48659d6002764413b1e1aaeed0a99e6a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 42521 } } peers { permanent_uuid: "003a69dc1f3b4de0961c0ef55d49d770" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33531 } }
I20260925 09:18:30.969096 21991 raft_consensus.cc:348] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4afb74ae9198452db84ada9cb748a394" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41577 } } peers { permanent_uuid: "48659d6002764413b1e1aaeed0a99e6a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 42521 } } peers { permanent_uuid: "003a69dc1f3b4de0961c0ef55d49d770" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33531 } }
I20260925 09:18:30.969580 21991 raft_consensus.cc:374] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:30.969694 21991 raft_consensus.cc:729] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48659d6002764413b1e1aaeed0a99e6a, State: Initialized, Role: FOLLOWER
I20260925 09:18:30.970214 21991 consensus_queue.cc:260] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4afb74ae9198452db84ada9cb748a394" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41577 } } peers { permanent_uuid: "48659d6002764413b1e1aaeed0a99e6a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 42521 } } peers { permanent_uuid: "003a69dc1f3b4de0961c0ef55d49d770" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33531 } }
I20260925 09:18:30.971208 21838 heartbeater.cc:503] Master 127.3.19.126:33395 was elected leader, sending a full tablet report...
I20260925 09:18:30.971855 21991 ts_tablet_manager.cc:1428] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 09:18:30.971896 21992 ts_tablet_manager.cc:1428] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260925 09:18:31.083659 21705 tablet.cc:2369] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:18:31.084301 21839 tablet.cc:2369] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:18:31.085052 21972 tablet.cc:2369] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:18:31.175040 21997 raft_consensus.cc:482] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:18:31.175400 21997 raft_consensus.cc:504] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4afb74ae9198452db84ada9cb748a394" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41577 } } peers { permanent_uuid: "48659d6002764413b1e1aaeed0a99e6a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 42521 } } peers { permanent_uuid: "003a69dc1f3b4de0961c0ef55d49d770" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33531 } }
I20260925 09:18:31.177302 21997 leader_election.cc:288] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4afb74ae9198452db84ada9cb748a394 (127.3.19.67:41577), 003a69dc1f3b4de0961c0ef55d49d770 (127.3.19.65:33531)
I20260925 09:18:31.184501 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:31.177710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:36851 (local address 127.3.19.67:41577)
0925 09:18:31.177945 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:31.177952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:31.178819 (+ 867us) server_negotiation.cc:407] Connection header received
0925 09:18:31.179025 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:31.179029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:31.179125 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:31.179213 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:31.179912 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.180708 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.182664 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.183541 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.183682 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:31.183788 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:31.184115 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 09:18:31.184237 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 09:18:31.184526 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:31.178015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:46167 (local address 127.3.19.65:33531)
0925 09:18:31.178243 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:31.178249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:31.178679 (+ 430us) server_negotiation.cc:407] Connection header received
0925 09:18:31.178939 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:31.178947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:31.179030 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:31.179098 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:31.179797 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.180544 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.182609 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.183508 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.183608 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:31.183756 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:31.184155 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 09:18:31.184277 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 09:18:31.184667 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:31.177692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:41577 (local address 127.3.19.66:36851)
0925 09:18:31.178707 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:31.178725 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:31.178878 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:31.179273 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:31.179277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:31.179320 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:31.179747 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:31.179761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.180843 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.180847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:31.182499 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:31.182511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.183776 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.183784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:31.183928 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:31.183981 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 09:18:31.184051 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 09:18:31.184193 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":45,"spinlock_wait_cycles":24448,"thread_start_us":715,"threads_started":1}
I20260925 09:18:31.184863 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:31.177551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:33531 (local address 127.3.19.66:46167)
0925 09:18:31.178540 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:31.178564 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:31.178727 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:31.179139 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:31.179144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:31.179171 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:31.179597 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:31.179617 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.180694 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.180699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:31.182465 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:31.182477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.183741 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.183748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:31.183928 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:31.184004 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 09:18:31.184110 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 09:18:31.184233 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":79,"thread_start_us":623,"threads_started":1}
I20260925 09:18:31.185925 21926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4c27e7cf554905b6b9d22280513cc3" candidate_uuid: "48659d6002764413b1e1aaeed0a99e6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4afb74ae9198452db84ada9cb748a394" is_pre_election: true
I20260925 09:18:31.185925 21659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4c27e7cf554905b6b9d22280513cc3" candidate_uuid: "48659d6002764413b1e1aaeed0a99e6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003a69dc1f3b4de0961c0ef55d49d770" is_pre_election: true
I20260925 09:18:31.186529 21926 raft_consensus.cc:2454] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48659d6002764413b1e1aaeed0a99e6a in term 0.
I20260925 09:18:31.186529 21659 raft_consensus.cc:2454] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48659d6002764413b1e1aaeed0a99e6a in term 0.
I20260925 09:18:31.187234 21726 leader_election.cc:302] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 003a69dc1f3b4de0961c0ef55d49d770, 48659d6002764413b1e1aaeed0a99e6a, 4afb74ae9198452db84ada9cb748a394; no voters:
I20260925 09:18:31.187647 21997 raft_consensus.cc:2781] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:18:31.187803 21997 raft_consensus.cc:482] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:18:31.187904 21997 raft_consensus.cc:3037] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:31.190111 21997 raft_consensus.cc:504] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4afb74ae9198452db84ada9cb748a394" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41577 } } peers { permanent_uuid: "48659d6002764413b1e1aaeed0a99e6a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 42521 } } peers { permanent_uuid: "003a69dc1f3b4de0961c0ef55d49d770" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33531 } }
I20260925 09:18:31.190711 21997 leader_election.cc:288] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [CANDIDATE]: Term 1 election: Requested vote from peers 4afb74ae9198452db84ada9cb748a394 (127.3.19.67:41577), 003a69dc1f3b4de0961c0ef55d49d770 (127.3.19.65:33531)
I20260925 09:18:31.191172 21926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4c27e7cf554905b6b9d22280513cc3" candidate_uuid: "48659d6002764413b1e1aaeed0a99e6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4afb74ae9198452db84ada9cb748a394"
I20260925 09:18:31.191226 21659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4c27e7cf554905b6b9d22280513cc3" candidate_uuid: "48659d6002764413b1e1aaeed0a99e6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003a69dc1f3b4de0961c0ef55d49d770"
I20260925 09:18:31.191386 21926 raft_consensus.cc:3037] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:31.191418 21659 raft_consensus.cc:3037] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:31.193720 21659 raft_consensus.cc:2454] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48659d6002764413b1e1aaeed0a99e6a in term 1.
I20260925 09:18:31.193730 21926 raft_consensus.cc:2454] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48659d6002764413b1e1aaeed0a99e6a in term 1.
I20260925 09:18:31.194072 21726 leader_election.cc:302] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003a69dc1f3b4de0961c0ef55d49d770, 48659d6002764413b1e1aaeed0a99e6a; no voters:
I20260925 09:18:31.194347 21997 raft_consensus.cc:2781] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:31.194810 21997 raft_consensus.cc:686] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [term 1 LEADER]: Becoming Leader. State: Replica: 48659d6002764413b1e1aaeed0a99e6a, State: Running, Role: LEADER
I20260925 09:18:31.195273 21997 consensus_queue.cc:237] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4afb74ae9198452db84ada9cb748a394" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41577 } } peers { permanent_uuid: "48659d6002764413b1e1aaeed0a99e6a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 42521 } } peers { permanent_uuid: "003a69dc1f3b4de0961c0ef55d49d770" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33531 } }
I20260925 09:18:31.199868 21515 catalog_manager.cc:5495] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 48659d6002764413b1e1aaeed0a99e6a (127.3.19.66). New cstate: current_term: 1 leader_uuid: "48659d6002764413b1e1aaeed0a99e6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4afb74ae9198452db84ada9cb748a394" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48659d6002764413b1e1aaeed0a99e6a" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 42521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "003a69dc1f3b4de0961c0ef55d49d770" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33531 } health_report { overall_health: UNKNOWN } } }
I20260925 09:18:31.227259 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:31.221871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.3.19.66:42521)
0925 09:18:31.222062 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:31.222084 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:31.222102 (+ 18us) server_negotiation.cc:407] Connection header received
0925 09:18:31.222317 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:31.222320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:31.222387 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:31.222556 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:18:31.222941 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.223637 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.224999 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.225522 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.225656 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:31.226217 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:18:31.226672 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:18:31.226848 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:31.226984 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 09:18:31.227027 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 09:18:31.234921 21659 raft_consensus.cc:1261] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770 [term 1 FOLLOWER]: Refusing update from remote peer 48659d6002764413b1e1aaeed0a99e6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:18:31.235132 21926 raft_consensus.cc:1261] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394 [term 1 FOLLOWER]: Refusing update from remote peer 48659d6002764413b1e1aaeed0a99e6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:18:31.235941 22000 consensus_queue.cc:1035] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "003a69dc1f3b4de0961c0ef55d49d770" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 33531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:18:31.236315 21997 consensus_queue.cc:1035] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4afb74ae9198452db84ada9cb748a394" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 41577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:18:31.247303 22003 mvcc.cc:205] Tried to move back new op lower bound from 7333183124408123392 to 7333183124259299328. Current Snapshot: MvccSnapshot[applied={T|T < 7333183124408123392}]
I20260925 09:18:31.272185 21515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:18:31.272505 21515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:18:31.276168 21515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8984ccd51d844f86af3be163fc574ff8: Sending DeleteTablet for 3 replicas of tablet 7b4c27e7cf554905b6b9d22280513cc3
I20260925 09:18:31.277349 21906 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b4c27e7cf554905b6b9d22280513cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:31 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:51310
I20260925 09:18:31.277436 21773 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b4c27e7cf554905b6b9d22280513cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:31 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:53962
I20260925 09:18:31.277671 21639 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b4c27e7cf554905b6b9d22280513cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:31 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:59574
I20260925 09:18:31.282631 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 21576
I20260925 09:18:31.288499 22013 tablet_replica.cc:330] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394: stopping tablet replica
I20260925 09:18:31.288955 22013 raft_consensus.cc:2229] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:18:31.290115 22014 tablet_replica.cc:330] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a: stopping tablet replica
I20260925 09:18:31.290530 22014 raft_consensus.cc:2229] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:18:31.291190 22014 raft_consensus.cc:2258] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:31.291920 22015 tablet_replica.cc:330] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770: stopping tablet replica
I20260925 09:18:31.292362 22015 raft_consensus.cc:2229] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:18:31.292899 22015 raft_consensus.cc:2258] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:31.295013 22015 ts_tablet_manager.cc:1905] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:31.297457 22013 raft_consensus.cc:2258] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:31.298418 22014 ts_tablet_manager.cc:1905] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 09:18:31 dist-test-slave-l6qh krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:18:31.301221 22013 ts_tablet_manager.cc:1905] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:31.305763 22014 ts_tablet_manager.cc:1918] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:31.306052 22013 ts_tablet_manager.cc:1918] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:31.306265 22013 log.cc:1196] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/7b4c27e7cf554905b6b9d22280513cc3
I20260925 09:18:31.306895 22013 ts_tablet_manager.cc:1939] T 7b4c27e7cf554905b6b9d22280513cc3 P 4afb74ae9198452db84ada9cb748a394: Deleting consensus metadata
I20260925 09:18:31.308910 21500 catalog_manager.cc:4841] TS 4afb74ae9198452db84ada9cb748a394 (127.3.19.67:41577): tablet 7b4c27e7cf554905b6b9d22280513cc3 (table test-table [id=de80731c6b8b4adc9beb173740e5be37]) successfully deleted
I20260925 09:18:31.311259 22014 log.cc:1196] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/7b4c27e7cf554905b6b9d22280513cc3
I20260925 09:18:31.311395 22015 ts_tablet_manager.cc:1918] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:31.311605 22015 log.cc:1196] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/7b4c27e7cf554905b6b9d22280513cc3
I20260925 09:18:31.311892 22014 ts_tablet_manager.cc:1939] T 7b4c27e7cf554905b6b9d22280513cc3 P 48659d6002764413b1e1aaeed0a99e6a: Deleting consensus metadata
I20260925 09:18:31.312235 22015 ts_tablet_manager.cc:1939] T 7b4c27e7cf554905b6b9d22280513cc3 P 003a69dc1f3b4de0961c0ef55d49d770: Deleting consensus metadata
I20260925 09:18:31.313652 21501 catalog_manager.cc:4841] TS 48659d6002764413b1e1aaeed0a99e6a (127.3.19.66:42521): tablet 7b4c27e7cf554905b6b9d22280513cc3 (table test-table [id=de80731c6b8b4adc9beb173740e5be37]) successfully deleted
I20260925 09:18:31.313951 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:31.284285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.3.19.65:33531)
0925 09:18:31.284532 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:31.284542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:31.291968 (+ 7426us) server_negotiation.cc:407] Connection header received
0925 09:18:31.292191 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:31.292195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:31.292262 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:31.292375 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:31.293972 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.294725 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.296115 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.296552 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.296699 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:31.306517 (+ 9818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:31.306538 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:31.306558 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:31.306611 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:31.310437 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:31.311146 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:31.311157 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:31.311162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:31.311405 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:31.311924 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:31.311929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:31.311932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:31.312407 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:31.312720 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:31.313312 (+ 592us) server_negotiation.cc:299] Negotiation successful
0925 09:18:31.313536 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 09:18:31.314105 21499 catalog_manager.cc:4841] TS 003a69dc1f3b4de0961c0ef55d49d770 (127.3.19.65:33531): tablet 7b4c27e7cf554905b6b9d22280513cc3 (table test-table [id=de80731c6b8b4adc9beb173740e5be37]) successfully deleted
I20260925 09:18:31.315135 21699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:31.435143 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 21576
I20260925 09:18:31.453083 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 21710
Sep 25 09:18:31 dist-test-slave-l6qh krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:18:31.480512 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:31.453949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.3.19.66:42521)
0925 09:18:31.454130 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:31.454134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:31.454152 (+ 18us) server_negotiation.cc:407] Connection header received
0925 09:18:31.454354 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:31.454358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:31.454429 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:31.454520 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:31.456135 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.456949 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.458992 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.459445 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.459539 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:31.467943 (+ 8404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:31.467966 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:31.467989 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:31.468037 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:31.475747 (+ 7710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:31.476823 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:31.476832 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:31.476836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:31.476996 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:31.477548 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:31.477558 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:31.477562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:31.478275 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:31.478595 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:31.479425 (+ 830us) server_negotiation.cc:299] Negotiation successful
0925 09:18:31.479662 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 09:18:31.484850 21833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:31.611357 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 21710
I20260925 09:18:31.630745 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 21843
Sep 25 09:18:31 dist-test-slave-l6qh krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:18:31.644923 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:31.631513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51322 (local address 127.3.19.67:41577)
0925 09:18:31.631693 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:31.631698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:31.631731 (+ 33us) server_negotiation.cc:407] Connection header received
0925 09:18:31.631848 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:31.631852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:31.631915 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:31.631995 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:31.633206 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.633989 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.635206 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:31.635576 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:31.635662 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:31.640029 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:31.640042 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:31.640058 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:31.640093 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:31.642690 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:31.643315 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:31.643322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:31.643324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:31.643452 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:31.643741 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:31.643744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:31.643746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:31.644060 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:31.644262 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:31.644492 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 09:18:31.644642 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 09:18:31.645642 21966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:31.739046 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 21843
I20260925 09:18:31.756771 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 21483
I20260925 09:18:31.757997 21545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:31.866544 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 21483
2026-09-25T09:18:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4416 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T09:18:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:31Z Disabled control of system clock
I20260925 09:18:31.944885 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:34969
--webserver_interface=127.3.19.126
--webserver_port=0
--builtin_ntp_servers=127.3.19.84:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:34969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 09:18:32.268358 22024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:32.268620 22024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:32.268726 22024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:32.275892 22024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:32.275991 22024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:32.276028 22024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:32.276078 22024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:32.276125 22024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:32.284825 22024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:34969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:34969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:32.286104 22024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:32.287771 22024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:32.299396 22032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:32.299466 22030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:32.299396 22029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:32.306799 22024 server_base.cc:1034] running on GCE node
I20260925 09:18:32.307610 22024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:32.308974 22024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:32.310207 22024 hybrid_clock.cc:648] HybridClock initialized: now 1790327912310159 us; error 67 us; skew 500 ppm
I20260925 09:18:32.310550 22024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:18:32.312750 22024 webserver.cc:460] Webserver started at http://127.3.19.126:37687/ using document root <none> and password file <none>
I20260925 09:18:32.313519 22024 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:32.313658 22024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:32.313998 22024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:32.316870 22024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "91a3acc2ba3a4b458c98999fb825bb3b"
format_stamp: "Formatted at 2026-09-25 09:18:32 on dist-test-slave-l6qh"
server_key: "e8df40a0ac0b01f37eddadae6c7a450d"
server_key_iv: "4ad5078962b38ad3b392daaef3c89cc0"
server_key_version: "encryptionkey@0"
I20260925 09:18:32.317641 22024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "91a3acc2ba3a4b458c98999fb825bb3b"
format_stamp: "Formatted at 2026-09-25 09:18:32 on dist-test-slave-l6qh"
server_key: "e8df40a0ac0b01f37eddadae6c7a450d"
server_key_iv: "4ad5078962b38ad3b392daaef3c89cc0"
server_key_version: "encryptionkey@0"
I20260925 09:18:32.322384 22024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 09:18:32.325598 22038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:32.326915 22024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:18:32.327100 22024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "91a3acc2ba3a4b458c98999fb825bb3b"
format_stamp: "Formatted at 2026-09-25 09:18:32 on dist-test-slave-l6qh"
server_key: "e8df40a0ac0b01f37eddadae6c7a450d"
server_key_iv: "4ad5078962b38ad3b392daaef3c89cc0"
server_key_version: "encryptionkey@0"
I20260925 09:18:32.327310 22024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:32.341183 22024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:32.341967 22024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:32.342243 22024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:32.362433 22024 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:34969
I20260925 09:18:32.362452 22089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:34969 every 8 connection(s)
I20260925 09:18:32.363754 22024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:32.366582 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 22024
I20260925 09:18:32.366750 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:32.367201 3149 external_mini_cluster.cc:1428] Setting key c2f56a8a86212bd954f7878446506f27
I20260925 09:18:32.367573 22090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:32.382543 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:32.369044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.3.19.126:34969)
0925 09:18:32.370127 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:32.370140 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:32.370188 (+ 48us) server_negotiation.cc:407] Connection header received
0925 09:18:32.372322 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:32.372350 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:32.372936 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:32.373423 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:32.373974 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:32.375969 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:32.377398 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:32.378459 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:32.378677 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:32.379804 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:32.379814 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:32.379831 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:32.380568 (+ 737us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:32.380671 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:32.380701 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:32.380805 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:32.380994 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 09:18:32.381325 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":399,"threads_started":1}
I20260925 09:18:32.382576 22090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b: Bootstrap starting.
I20260925 09:18:32.385807 22090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:32.386775 22090 log.cc:824] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:32.389439 22090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b: No bootstrap required, opened a new log
I20260925 09:18:32.396674 22090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a3acc2ba3a4b458c98999fb825bb3b" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34969 } }
I20260925 09:18:32.396970 22090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:32.397060 22090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a3acc2ba3a4b458c98999fb825bb3b, State: Initialized, Role: FOLLOWER
I20260925 09:18:32.397648 22090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a3acc2ba3a4b458c98999fb825bb3b" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34969 } }
I20260925 09:18:32.397854 22090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:32.397969 22090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:32.398087 22090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:32.400286 22090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a3acc2ba3a4b458c98999fb825bb3b" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34969 } }
I20260925 09:18:32.400797 22090 leader_election.cc:302] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91a3acc2ba3a4b458c98999fb825bb3b; no voters:
I20260925 09:18:32.401383 22090 leader_election.cc:288] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:32.401530 22095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:32.402268 22095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [term 1 LEADER]: Becoming Leader. State: Replica: 91a3acc2ba3a4b458c98999fb825bb3b, State: Running, Role: LEADER
I20260925 09:18:32.402781 22095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a3acc2ba3a4b458c98999fb825bb3b" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34969 } }
I20260925 09:18:32.403331 22090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:32.405740 22097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91a3acc2ba3a4b458c98999fb825bb3b. Latest consensus state: current_term: 1 leader_uuid: "91a3acc2ba3a4b458c98999fb825bb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a3acc2ba3a4b458c98999fb825bb3b" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34969 } } }
I20260925 09:18:32.405828 22096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91a3acc2ba3a4b458c98999fb825bb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a3acc2ba3a4b458c98999fb825bb3b" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34969 } } }
I20260925 09:18:32.406484 22096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:32.406490 22097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:32.410468 22103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:32.415423 22103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:32.423556 22103 catalog_manager.cc:1347] Generated new cluster ID: 59a9525bb79a4902bad0cca39c297791
I20260925 09:18:32.423677 22103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:32.443629 22103 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:32.445374 22103 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:32.454186 22103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b: Generated new TSK 0
I20260925 09:18:32.455114 22103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:18:32.466485 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:34969
--builtin_ntp_servers=127.3.19.84:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 09:18:32.792265 22114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:32.792524 22114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:32.792613 22114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:32.799595 22114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:32.799682 22114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:32.799779 22114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:18:32.808338 22114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:34969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:32.809494 22114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:32.811364 22114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:32.824344 22120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:32.824429 22119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:32.824455 22122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:32.825307 22114 server_base.cc:1034] running on GCE node
I20260925 09:18:32.826004 22114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:32.826932 22114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:32.828166 22114 hybrid_clock.cc:648] HybridClock initialized: now 1790327912828122 us; error 53 us; skew 500 ppm
I20260925 09:18:32.828498 22114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:18:32.830958 22114 webserver.cc:460] Webserver started at http://127.3.19.65:45977/ using document root <none> and password file <none>
I20260925 09:18:32.832021 22114 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:32.832162 22114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:32.832489 22114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:32.835467 22114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "2b584d594d2f46e8b6ee8ac4022b895e"
format_stamp: "Formatted at 2026-09-25 09:18:32 on dist-test-slave-l6qh"
server_key: "47867c39405d9fa47f3fbdef4ba17660"
server_key_iv: "3bfb4714e9e09b299933bf6b6c826e23"
server_key_version: "encryptionkey@0"
I20260925 09:18:32.836323 22114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "2b584d594d2f46e8b6ee8ac4022b895e"
format_stamp: "Formatted at 2026-09-25 09:18:32 on dist-test-slave-l6qh"
server_key: "47867c39405d9fa47f3fbdef4ba17660"
server_key_iv: "3bfb4714e9e09b299933bf6b6c826e23"
server_key_version: "encryptionkey@0"
I20260925 09:18:32.841488 22114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 09:18:32.844956 22128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:32.846441 22114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:18:32.846659 22114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "2b584d594d2f46e8b6ee8ac4022b895e"
format_stamp: "Formatted at 2026-09-25 09:18:32 on dist-test-slave-l6qh"
server_key: "47867c39405d9fa47f3fbdef4ba17660"
server_key_iv: "3bfb4714e9e09b299933bf6b6c826e23"
server_key_version: "encryptionkey@0"
I20260925 09:18:32.846894 22114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:32.863261 22114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:32.864049 22114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:32.864323 22114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:32.865319 22114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:32.866726 22114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:32.866849 22114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:32.866983 22114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:32.867074 22114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:32.900771 22114 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:36387
I20260925 09:18:32.900826 22240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:36387 every 8 connection(s)
I20260925 09:18:32.902160 22114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:18:32.910573 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 22114
I20260925 09:18:32.910759 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:18:32.911211 3149 external_mini_cluster.cc:1428] Setting key 6dac56136a77b58e551597c5618b5c4a
I20260925 09:18:32.913899 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:34969
--builtin_ntp_servers=127.3.19.84:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 09:18:32.915366 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:32.904854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:43019 (local address 127.3.19.126:34969)
0925 09:18:32.905348 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:32.905352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:32.906493 (+ 1141us) server_negotiation.cc:407] Connection header received
0925 09:18:32.908223 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:32.908228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:32.908314 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:32.908430 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:32.909489 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:32.910374 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:32.912392 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:32.912966 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:32.913107 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:32.914545 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:32.914549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:32.914552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:32.914569 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:32.914611 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:32.914623 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:32.914717 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:32.914941 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 09:18:32.915050 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":259,"threads_started":1}
I20260925 09:18:32.916437 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:32.905477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:34969 (local address 127.3.19.65:43019)
0925 09:18:32.906334 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:32.906381 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:32.907954 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:32.908662 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:32.908670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:32.908751 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:32.909334 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:32.909352 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:32.910693 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:32.910701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:32.912260 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:32.912273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:32.913196 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:32.913205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:32.913468 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:32.914241 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:18:32.914243 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:32.914280 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:18:32.914348 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:32.914716 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:32.914729 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:32.914739 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 09:18:32.914910 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 09:18:32.915304 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":259,"threads_started":1}
I20260925 09:18:32.917922 22241 heartbeater.cc:347] Connected to a master server at 127.3.19.126:34969
I20260925 09:18:32.918210 22241 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:32.918929 22241 heartbeater.cc:511] Master 127.3.19.126:34969 requested a full tablet report, sending...
I20260925 09:18:32.921077 22055 ts_manager.cc:194] Registered new tserver with Master: 2b584d594d2f46e8b6ee8ac4022b895e (127.3.19.65:36387)
I20260925 09:18:32.923261 22055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.19.65:43019
W20260925 09:18:33.235514 22245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:33.235777 22245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:33.235910 22245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:33.242929 22245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:33.243014 22245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:33.243172 22245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:18:33.251922 22245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:34969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:33.253121 22245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:33.254804 22245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:33.267247 22253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:33.267283 22251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:33.267210 22250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:33.268321 22245 server_base.cc:1034] running on GCE node
I20260925 09:18:33.268877 22245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:33.269835 22245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:33.271026 22245 hybrid_clock.cc:648] HybridClock initialized: now 1790327913270995 us; error 48 us; skew 500 ppm
I20260925 09:18:33.271343 22245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:18:33.273471 22245 webserver.cc:460] Webserver started at http://127.3.19.66:43367/ using document root <none> and password file <none>
I20260925 09:18:33.274113 22245 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:33.274223 22245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:33.274531 22245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:33.277196 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "72e5b88e4be743bc8d8b4a150a22e42e"
format_stamp: "Formatted at 2026-09-25 09:18:33 on dist-test-slave-l6qh"
server_key: "369d2f615f3e32d80000a70a98fb7f22"
server_key_iv: "5b274875067289ed7ac4fb6ca1537f4b"
server_key_version: "encryptionkey@0"
I20260925 09:18:33.277977 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "72e5b88e4be743bc8d8b4a150a22e42e"
format_stamp: "Formatted at 2026-09-25 09:18:33 on dist-test-slave-l6qh"
server_key: "369d2f615f3e32d80000a70a98fb7f22"
server_key_iv: "5b274875067289ed7ac4fb6ca1537f4b"
server_key_version: "encryptionkey@0"
I20260925 09:18:33.282841 22245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 09:18:33.286154 22259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:33.287355 22245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 09:18:33.287523 22245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "72e5b88e4be743bc8d8b4a150a22e42e"
format_stamp: "Formatted at 2026-09-25 09:18:33 on dist-test-slave-l6qh"
server_key: "369d2f615f3e32d80000a70a98fb7f22"
server_key_iv: "5b274875067289ed7ac4fb6ca1537f4b"
server_key_version: "encryptionkey@0"
I20260925 09:18:33.287740 22245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:33.303455 22245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:33.304214 22245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:33.304471 22245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:33.305408 22245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:33.306738 22245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:33.306849 22245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:33.306986 22245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:33.307082 22245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:33.339039 22245 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:33095
I20260925 09:18:33.339092 22371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:33095 every 8 connection(s)
I20260925 09:18:33.340313 22245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:18:33.340716 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 22245
I20260925 09:18:33.340912 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:18:33.341569 3149 external_mini_cluster.cc:1428] Setting key 1cb7054b751418f22a2a8d20b2d15508
I20260925 09:18:33.344591 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:34969
--builtin_ntp_servers=127.3.19.84:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 09:18:33.354543 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:33.343057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:42031 (local address 127.3.19.126:34969)
0925 09:18:33.343268 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:33.343275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:33.344899 (+ 1624us) server_negotiation.cc:407] Connection header received
0925 09:18:33.347684 (+ 2785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:33.347688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:33.347751 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:33.347831 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:33.349187 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.350065 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.351829 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.352217 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.352351 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:33.353700 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:33.353705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:33.353708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:33.353725 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:33.353756 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:33.353789 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:33.353921 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:33.354150 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 09:18:33.354241 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 09:18:33.355515 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:33.343678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:34969 (local address 127.3.19.66:42031)
0925 09:18:33.344729 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:33.344782 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:33.347390 (+ 2608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:33.348090 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:33.348111 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:33.348223 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:33.348979 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:33.349004 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.350253 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.350265 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:33.351684 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:33.351697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.352433 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.352440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:33.352670 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:33.353393 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:18:33.353396 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:33.353432 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:18:33.353507 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:33.353937 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:33.353949 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:33.353963 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 09:18:33.354139 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 09:18:33.354513 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":293,"threads_started":1}
I20260925 09:18:33.356967 22372 heartbeater.cc:347] Connected to a master server at 127.3.19.126:34969
I20260925 09:18:33.357317 22372 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:33.358021 22372 heartbeater.cc:511] Master 127.3.19.126:34969 requested a full tablet report, sending...
I20260925 09:18:33.359470 22055 ts_manager.cc:194] Registered new tserver with Master: 72e5b88e4be743bc8d8b4a150a22e42e (127.3.19.66:33095)
I20260925 09:18:33.360718 22055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.19.66:42031
W20260925 09:18:33.679795 22375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:33.680073 22375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:33.680163 22375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:33.687477 22375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:33.687635 22375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:33.687868 22375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:18:33.697309 22375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:34969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:33.698632 22375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:33.700587 22375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:33.713503 22380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:33.713585 22381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:33.713598 22383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:33.714447 22375 server_base.cc:1034] running on GCE node
I20260925 09:18:33.715008 22375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:33.715854 22375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:33.717095 22375 hybrid_clock.cc:648] HybridClock initialized: now 1790327913717050 us; error 54 us; skew 500 ppm
I20260925 09:18:33.717432 22375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:18:33.719595 22375 webserver.cc:460] Webserver started at http://127.3.19.67:40483/ using document root <none> and password file <none>
I20260925 09:18:33.720258 22375 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:33.720386 22375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:33.720703 22375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:33.723423 22375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "3e161600abfa422d94b9de50b3720bf5"
format_stamp: "Formatted at 2026-09-25 09:18:33 on dist-test-slave-l6qh"
server_key: "704657fb4f707b91354157be823809ed"
server_key_iv: "dd76416bd272cf5ce189cacca5587f4a"
server_key_version: "encryptionkey@0"
I20260925 09:18:33.724166 22375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "3e161600abfa422d94b9de50b3720bf5"
format_stamp: "Formatted at 2026-09-25 09:18:33 on dist-test-slave-l6qh"
server_key: "704657fb4f707b91354157be823809ed"
server_key_iv: "dd76416bd272cf5ce189cacca5587f4a"
server_key_version: "encryptionkey@0"
I20260925 09:18:33.729117 22375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 09:18:33.732757 22389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:33.734114 22375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:18:33.734293 22375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "3e161600abfa422d94b9de50b3720bf5"
format_stamp: "Formatted at 2026-09-25 09:18:33 on dist-test-slave-l6qh"
server_key: "704657fb4f707b91354157be823809ed"
server_key_iv: "dd76416bd272cf5ce189cacca5587f4a"
server_key_version: "encryptionkey@0"
I20260925 09:18:33.734516 22375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:33.761986 22375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:33.762832 22375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:33.763135 22375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:33.764086 22375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:33.765655 22375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:33.765793 22375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:33.765955 22375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:33.766050 22375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:33.799906 22375 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:39491
I20260925 09:18:33.799991 22501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:39491 every 8 connection(s)
I20260925 09:18:33.801302 22375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:18:33.804422 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 22375
I20260925 09:18:33.804595 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:18:33.805264 3149 external_mini_cluster.cc:1428] Setting key 5a6c7dd1655a51bb1f6b7d94a81223c7
I20260925 09:18:33.813963 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:33.803832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:37951 (local address 127.3.19.126:34969)
0925 09:18:33.804021 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:33.804028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:33.805818 (+ 1790us) server_negotiation.cc:407] Connection header received
0925 09:18:33.808018 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:33.808024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:33.808110 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:33.808188 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:33.809174 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.809955 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.811494 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.811861 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.811973 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:33.813206 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:33.813211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:33.813214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:33.813293 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:33.813328 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:33.813342 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:33.813418 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:33.813641 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 09:18:33.813716 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 09:18:33.814921 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:33.804435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:34969 (local address 127.3.19.67:37951)
0925 09:18:33.805642 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:33.805689 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:33.807691 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:33.808393 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:33.808400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:33.808481 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:33.809005 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:33.809024 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.810065 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.810069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:33.811394 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:33.811405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.812057 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.812061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:33.812279 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:33.812928 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:18:33.812930 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:33.812969 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:18:33.813038 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:33.813428 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:33.813436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:33.813441 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 09:18:33.813581 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 09:18:33.813864 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":319,"threads_started":1}
I20260925 09:18:33.816190 22502 heartbeater.cc:347] Connected to a master server at 127.3.19.126:34969
I20260925 09:18:33.816471 22502 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:33.817124 22502 heartbeater.cc:511] Master 127.3.19.126:34969 requested a full tablet report, sending...
I20260925 09:18:33.818574 22055 ts_manager.cc:194] Registered new tserver with Master: 3e161600abfa422d94b9de50b3720bf5 (127.3.19.67:39491)
I20260925 09:18:33.819581 22055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.19.67:37951
I20260925 09:18:33.820323 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:18:33.830479 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:33.824738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.3.19.126:34969)
0925 09:18:33.824935 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:33.824945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:33.825426 (+ 481us) server_negotiation.cc:407] Connection header received
0925 09:18:33.825526 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:33.825529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:33.825594 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:33.825680 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:33.826116 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.826779 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.828182 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.828552 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.828661 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:33.829761 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:33.829765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:33.829767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:33.829783 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:33.829823 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:33.829833 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:33.829934 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:33.830079 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 09:18:33.830119 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 09:18:33.834138 22054 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:18:33.840747 22054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:18:33.865298 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:33.852557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:33095 (local address 127.0.0.1:59872)
0925 09:18:33.855713 (+ 3156us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:33.855738 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:33.855844 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:33.857045 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:33.857056 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:33.857121 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:33.857683 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:33.857699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.860037 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.860045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:33.862378 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:33.862389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.864310 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.864317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:33.864484 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:33.864543 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 09:18:33.864664 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 09:18:33.864858 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3012,"mutex_wait_us":45,"spinlock_wait_cycles":6912,"thread_start_us":1596,"threads_started":1}
I20260925 09:18:33.865648 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:33.852433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.3.19.65:36387)
0925 09:18:33.856467 (+ 4034us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:33.856483 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:33.856511 (+ 28us) server_negotiation.cc:407] Connection header received
0925 09:18:33.856607 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:33.856616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:33.856851 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:33.857041 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:33.857712 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.858724 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.862619 (+ 3895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.864073 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.864217 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:33.864489 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:33.865074 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 09:18:33.865278 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3847,"thread_start_us":2642,"threads_started":1}
I20260925 09:18:33.865474 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:33.852264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:36387 (local address 127.0.0.1:39782)
0925 09:18:33.854493 (+ 2229us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:33.854539 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:33.854705 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:33.857043 (+ 2338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:33.857050 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:33.857098 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:33.857531 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:33.857540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.858848 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.858856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:33.862347 (+ 3491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:33.862367 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.864673 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.864687 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:33.864870 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:33.864898 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 09:18:33.864975 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 09:18:33.865085 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2006,"mutex_wait_us":6,"spinlock_wait_cycles":2432,"thread_start_us":3163,"threads_started":1}
I20260925 09:18:33.866575 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:33.853274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.3.19.66:33095)
0925 09:18:33.856355 (+ 3081us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:33.856369 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:33.856403 (+ 34us) server_negotiation.cc:407] Connection header received
0925 09:18:33.856535 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:33.856545 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:33.856838 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:33.857894 (+ 1056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:33.858127 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.859912 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.862639 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.864073 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.865573 (+ 1500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:33.865804 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:33.865893 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 09:18:33.866348 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2896,"thread_start_us":2775,"threads_started":1}
I20260925 09:18:33.866684 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:33.857013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:39491 (local address 127.0.0.1:47234)
0925 09:18:33.859315 (+ 2302us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:33.859337 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:33.859445 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:33.860546 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:33.860554 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:33.860578 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:33.861031 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:33.861043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.862346 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.862353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:33.863957 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:33.863970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.865932 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.865938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:33.866134 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:33.866194 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 09:18:33.866287 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 09:18:33.866403 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2158,"thread_start_us":2001,"threads_started":1}
I20260925 09:18:33.867385 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:33.856139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.3.19.67:39491)
0925 09:18:33.859977 (+ 3838us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:33.859985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:33.860016 (+ 31us) server_negotiation.cc:407] Connection header received
0925 09:18:33.860114 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:33.860121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:33.860354 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:33.860558 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:33.861219 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.862204 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.864182 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:33.865397 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:33.865517 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:33.865809 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:33.866999 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0925 09:18:33.867131 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3676,"thread_start_us":3522,"threads_started":1}
I20260925 09:18:33.869637 22176 tablet_service.cc:1467] Processing CreateTablet for tablet dee7242612f44efe963f8059e86b15ab (DEFAULT_TABLE table=test-table [id=7d78821ffd3a493cb13d674920da1f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:33.869859 22307 tablet_service.cc:1467] Processing CreateTablet for tablet dee7242612f44efe963f8059e86b15ab (DEFAULT_TABLE table=test-table [id=7d78821ffd3a493cb13d674920da1f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:33.870885 22176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee7242612f44efe963f8059e86b15ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:33.871212 22307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee7242612f44efe963f8059e86b15ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:33.872030 22437 tablet_service.cc:1467] Processing CreateTablet for tablet dee7242612f44efe963f8059e86b15ab (DEFAULT_TABLE table=test-table [id=7d78821ffd3a493cb13d674920da1f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:33.873394 22437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee7242612f44efe963f8059e86b15ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:33.883137 22520 tablet_bootstrap.cc:492] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e: Bootstrap starting.
I20260925 09:18:33.885262 22521 tablet_bootstrap.cc:492] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e: Bootstrap starting.
I20260925 09:18:33.886411 22520 tablet_bootstrap.cc:654] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:33.886962 22522 tablet_bootstrap.cc:492] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5: Bootstrap starting.
I20260925 09:18:33.887706 22520 log.cc:824] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:33.888908 22521 tablet_bootstrap.cc:654] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:33.890043 22521 log.cc:824] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:33.890163 22522 tablet_bootstrap.cc:654] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:33.891453 22522 log.cc:824] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:33.891453 22520 tablet_bootstrap.cc:492] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e: No bootstrap required, opened a new log
I20260925 09:18:33.891897 22520 ts_tablet_manager.cc:1397] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 09:18:33.895569 22521 tablet_bootstrap.cc:492] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e: No bootstrap required, opened a new log
I20260925 09:18:33.895840 22521 ts_tablet_manager.cc:1397] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260925 09:18:33.895843 22522 tablet_bootstrap.cc:492] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5: No bootstrap required, opened a new log
I20260925 09:18:33.896111 22522 ts_tablet_manager.cc:1397] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260925 09:18:33.903781 22520 raft_consensus.cc:348] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36387 } } peers { permanent_uuid: "72e5b88e4be743bc8d8b4a150a22e42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33095 } } peers { permanent_uuid: "3e161600abfa422d94b9de50b3720bf5" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39491 } }
I20260925 09:18:33.904449 22520 raft_consensus.cc:374] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:33.904610 22520 raft_consensus.cc:729] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b584d594d2f46e8b6ee8ac4022b895e, State: Initialized, Role: FOLLOWER
I20260925 09:18:33.904707 22521 raft_consensus.cc:348] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36387 } } peers { permanent_uuid: "72e5b88e4be743bc8d8b4a150a22e42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33095 } } peers { permanent_uuid: "3e161600abfa422d94b9de50b3720bf5" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39491 } }
I20260925 09:18:33.905191 22521 raft_consensus.cc:374] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:33.905373 22521 raft_consensus.cc:729] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e5b88e4be743bc8d8b4a150a22e42e, State: Initialized, Role: FOLLOWER
I20260925 09:18:33.905457 22520 consensus_queue.cc:260] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36387 } } peers { permanent_uuid: "72e5b88e4be743bc8d8b4a150a22e42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33095 } } peers { permanent_uuid: "3e161600abfa422d94b9de50b3720bf5" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39491 } }
I20260925 09:18:33.905779 22522 raft_consensus.cc:348] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36387 } } peers { permanent_uuid: "72e5b88e4be743bc8d8b4a150a22e42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33095 } } peers { permanent_uuid: "3e161600abfa422d94b9de50b3720bf5" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39491 } }
I20260925 09:18:33.906225 22522 raft_consensus.cc:374] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:33.906347 22522 raft_consensus.cc:729] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e161600abfa422d94b9de50b3720bf5, State: Initialized, Role: FOLLOWER
I20260925 09:18:33.906067 22521 consensus_queue.cc:260] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36387 } } peers { permanent_uuid: "72e5b88e4be743bc8d8b4a150a22e42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33095 } } peers { permanent_uuid: "3e161600abfa422d94b9de50b3720bf5" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39491 } }
I20260925 09:18:33.906992 22522 consensus_queue.cc:260] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36387 } } peers { permanent_uuid: "72e5b88e4be743bc8d8b4a150a22e42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33095 } } peers { permanent_uuid: "3e161600abfa422d94b9de50b3720bf5" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39491 } }
I20260925 09:18:33.912037 22522 ts_tablet_manager.cc:1428] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20260925 09:18:33.912401 22502 heartbeater.cc:503] Master 127.3.19.126:34969 was elected leader, sending a full tablet report...
I20260925 09:18:33.914697 22372 heartbeater.cc:503] Master 127.3.19.126:34969 was elected leader, sending a full tablet report...
I20260925 09:18:33.915638 22521 ts_tablet_manager.cc:1428] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260925 09:18:33.922154 22241 heartbeater.cc:503] Master 127.3.19.126:34969 was elected leader, sending a full tablet report...
I20260925 09:18:33.922735 22520 ts_tablet_manager.cc:1428] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e: Time spent starting tablet: real 0.031s user 0.016s sys 0.012s
I20260925 09:18:34.048885 22526 raft_consensus.cc:482] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:18:34.049274 22526 raft_consensus.cc:504] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36387 } } peers { permanent_uuid: "72e5b88e4be743bc8d8b4a150a22e42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33095 } } peers { permanent_uuid: "3e161600abfa422d94b9de50b3720bf5" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39491 } }
I20260925 09:18:34.051136 22526 leader_election.cc:288] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72e5b88e4be743bc8d8b4a150a22e42e (127.3.19.66:33095), 3e161600abfa422d94b9de50b3720bf5 (127.3.19.67:39491)
W20260925 09:18:34.053018 22503 tablet.cc:2369] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:18:34.058089 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:34.051466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:45819 (local address 127.3.19.66:33095)
0925 09:18:34.051698 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:34.051708 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:34.052473 (+ 765us) server_negotiation.cc:407] Connection header received
0925 09:18:34.052664 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:34.052668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:34.052734 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:34.052837 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:34.053696 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.054367 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.056364 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.057092 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.057236 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:34.057365 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:34.057690 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 09:18:34.057818 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 09:18:34.058138 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:34.051721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:35037 (local address 127.3.19.67:39491)
0925 09:18:34.051917 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:34.051923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:34.052319 (+ 396us) server_negotiation.cc:407] Connection header received
0925 09:18:34.052581 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:34.052590 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:34.052675 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:34.052797 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:34.053537 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.054403 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.056358 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.057109 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.057265 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:34.057384 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:34.057717 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 09:18:34.057883 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 09:18:34.058251 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:34.051477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:33095 (local address 127.3.19.65:45819)
0925 09:18:34.052372 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:34.052386 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:34.052527 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:34.053003 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:34.053008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:34.053042 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:34.053533 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:34.053543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.054492 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.054499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:34.056155 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:34.056179 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.057307 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.057314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:34.057495 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:34.057555 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 09:18:34.057658 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 09:18:34.057786 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":112,"thread_start_us":523,"threads_started":1}
I20260925 09:18:34.058416 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:34.051447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:39491 (local address 127.3.19.65:35037)
0925 09:18:34.052217 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:34.052231 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:34.052399 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:34.052838 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:34.052843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:34.052880 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:34.053344 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:34.053380 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.054544 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.054554 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:34.056151 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:34.056175 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.057371 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.057378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:34.057565 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:34.057586 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 09:18:34.057691 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 09:18:34.057805 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":72,"spinlock_wait_cycles":19712,"thread_start_us":479,"threads_started":1}
I20260925 09:18:34.059165 22327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee7242612f44efe963f8059e86b15ab" candidate_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e5b88e4be743bc8d8b4a150a22e42e" is_pre_election: true
I20260925 09:18:34.059234 22457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee7242612f44efe963f8059e86b15ab" candidate_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e161600abfa422d94b9de50b3720bf5" is_pre_election: true
I20260925 09:18:34.059718 22327 raft_consensus.cc:2454] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b584d594d2f46e8b6ee8ac4022b895e in term 0.
I20260925 09:18:34.059752 22457 raft_consensus.cc:2454] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b584d594d2f46e8b6ee8ac4022b895e in term 0.
I20260925 09:18:34.060379 22131 leader_election.cc:302] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b584d594d2f46e8b6ee8ac4022b895e, 3e161600abfa422d94b9de50b3720bf5, 72e5b88e4be743bc8d8b4a150a22e42e; no voters:
I20260925 09:18:34.060778 22526 raft_consensus.cc:2781] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:18:34.060911 22526 raft_consensus.cc:482] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:18:34.061023 22526 raft_consensus.cc:3037] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:34.063216 22526 raft_consensus.cc:504] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36387 } } peers { permanent_uuid: "72e5b88e4be743bc8d8b4a150a22e42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33095 } } peers { permanent_uuid: "3e161600abfa422d94b9de50b3720bf5" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39491 } }
I20260925 09:18:34.063783 22526 leader_election.cc:288] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [CANDIDATE]: Term 1 election: Requested vote from peers 72e5b88e4be743bc8d8b4a150a22e42e (127.3.19.66:33095), 3e161600abfa422d94b9de50b3720bf5 (127.3.19.67:39491)
I20260925 09:18:34.064195 22327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee7242612f44efe963f8059e86b15ab" candidate_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e5b88e4be743bc8d8b4a150a22e42e"
I20260925 09:18:34.064225 22457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee7242612f44efe963f8059e86b15ab" candidate_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e161600abfa422d94b9de50b3720bf5"
I20260925 09:18:34.064388 22327 raft_consensus.cc:3037] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:34.064395 22457 raft_consensus.cc:3037] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:34.066586 22457 raft_consensus.cc:2454] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b584d594d2f46e8b6ee8ac4022b895e in term 1.
I20260925 09:18:34.066695 22327 raft_consensus.cc:2454] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b584d594d2f46e8b6ee8ac4022b895e in term 1.
I20260925 09:18:34.067044 22132 leader_election.cc:302] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b584d594d2f46e8b6ee8ac4022b895e, 3e161600abfa422d94b9de50b3720bf5; no voters:
I20260925 09:18:34.067363 22526 raft_consensus.cc:2781] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:34.067822 22526 raft_consensus.cc:686] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [term 1 LEADER]: Becoming Leader. State: Replica: 2b584d594d2f46e8b6ee8ac4022b895e, State: Running, Role: LEADER
I20260925 09:18:34.068351 22526 consensus_queue.cc:237] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36387 } } peers { permanent_uuid: "72e5b88e4be743bc8d8b4a150a22e42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33095 } } peers { permanent_uuid: "3e161600abfa422d94b9de50b3720bf5" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39491 } }
I20260925 09:18:34.072490 22055 catalog_manager.cc:5495] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b584d594d2f46e8b6ee8ac4022b895e (127.3.19.65). New cstate: current_term: 1 leader_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b584d594d2f46e8b6ee8ac4022b895e" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 36387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72e5b88e4be743bc8d8b4a150a22e42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e161600abfa422d94b9de50b3720bf5" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39491 } health_report { overall_health: UNKNOWN } } }
W20260925 09:18:34.092374 22373 tablet.cc:2369] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:18:34.101390 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:34.096428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39796 (local address 127.3.19.65:36387)
0925 09:18:34.096573 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:34.096577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:34.096592 (+ 15us) server_negotiation.cc:407] Connection header received
0925 09:18:34.096753 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:34.096760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:34.096880 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:34.097026 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:18:34.097461 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.098105 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.099441 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.099820 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.099912 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:34.100335 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:18:34.100796 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:18:34.100911 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:34.101008 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 09:18:34.101064 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 09:18:34.109179 22457 raft_consensus.cc:1261] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5 [term 1 FOLLOWER]: Refusing update from remote peer 2b584d594d2f46e8b6ee8ac4022b895e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:18:34.109174 22327 raft_consensus.cc:1261] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e [term 1 FOLLOWER]: Refusing update from remote peer 2b584d594d2f46e8b6ee8ac4022b895e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:18:34.110213 22531 consensus_queue.cc:1035] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e5b88e4be743bc8d8b4a150a22e42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:18:34.110549 22526 consensus_queue.cc:1035] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e161600abfa422d94b9de50b3720bf5" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 39491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:18:34.120782 22534 mvcc.cc:205] Tried to move back new op lower bound from 7333183136181530624 to 7333183136027611136. Current Snapshot: MvccSnapshot[applied={T|T < 7333183136181530624}]
I20260925 09:18:34.121956 22535 mvcc.cc:205] Tried to move back new op lower bound from 7333183136181530624 to 7333183136027611136. Current Snapshot: MvccSnapshot[applied={T|T < 7333183136181530624}]
I20260925 09:18:34.135464 22054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:18:34.135766 22054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:18:34.139477 22054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91a3acc2ba3a4b458c98999fb825bb3b: Sending DeleteTablet for 3 replicas of tablet dee7242612f44efe963f8059e86b15ab
I20260925 09:18:34.140501 22176 tablet_service.cc:1514] Processing DeleteTablet for tablet dee7242612f44efe963f8059e86b15ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:34 UTC) from {username='slave'} at 127.0.0.1:39782
I20260925 09:18:34.140616 22307 tablet_service.cc:1514] Processing DeleteTablet for tablet dee7242612f44efe963f8059e86b15ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:34 UTC) from {username='slave'} at 127.0.0.1:59872
I20260925 09:18:34.140839 22437 tablet_service.cc:1514] Processing DeleteTablet for tablet dee7242612f44efe963f8059e86b15ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:34 UTC) from {username='slave'} at 127.0.0.1:47234
I20260925 09:18:34.142709 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 22114
I20260925 09:18:34.152894 22545 tablet_replica.cc:330] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e: stopping tablet replica
I20260925 09:18:34.153375 22544 tablet_replica.cc:330] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e: stopping tablet replica
I20260925 09:18:34.153532 22545 raft_consensus.cc:2229] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:18:34.154160 22545 raft_consensus.cc:2258] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:34.155194 22544 raft_consensus.cc:2229] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:18:34.156248 22544 raft_consensus.cc:2258] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:34.156766 22545 ts_tablet_manager.cc:1905] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:34.158891 22544 ts_tablet_manager.cc:1905] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:34.162477 22546 tablet_replica.cc:330] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5: stopping tablet replica
I20260925 09:18:34.164297 22545 ts_tablet_manager.cc:1918] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:34.164506 22545 log.cc:1196] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/dee7242612f44efe963f8059e86b15ab
I20260925 09:18:34.164567 22546 raft_consensus.cc:2229] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:18:34.165074 22546 raft_consensus.cc:2258] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:34.165297 22545 ts_tablet_manager.cc:1939] T dee7242612f44efe963f8059e86b15ab P 72e5b88e4be743bc8d8b4a150a22e42e: Deleting consensus metadata
I20260925 09:18:34.167275 22041 catalog_manager.cc:4841] TS 72e5b88e4be743bc8d8b4a150a22e42e (127.3.19.66:33095): tablet dee7242612f44efe963f8059e86b15ab (table test-table [id=7d78821ffd3a493cb13d674920da1f02]) successfully deleted
I20260925 09:18:34.167626 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:34.144066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.3.19.65:36387)
0925 09:18:34.144232 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:34.144238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:34.157772 (+ 13534us) server_negotiation.cc:407] Connection header received
0925 09:18:34.158162 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:34.158167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:34.158261 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:34.158331 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:34.158940 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.160143 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.162687 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.163363 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.163466 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:34.165834 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:34.165845 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:34.165862 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:34.166706 (+ 844us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:34.166776 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:34.166802 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:34.166895 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:34.167140 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 09:18:34.167248 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 09:18:34.168000 22544 ts_tablet_manager.cc:1918] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:34.168164 22544 log.cc:1196] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/dee7242612f44efe963f8059e86b15ab
I20260925 09:18:34.168891 22544 ts_tablet_manager.cc:1939] T dee7242612f44efe963f8059e86b15ab P 2b584d594d2f46e8b6ee8ac4022b895e: Deleting consensus metadata
I20260925 09:18:34.168915 22236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:34.169947 22546 ts_tablet_manager.cc:1905] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:34.176080 22546 ts_tablet_manager.cc:1918] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:34.176231 22546 log.cc:1196] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/dee7242612f44efe963f8059e86b15ab
I20260925 09:18:34.176687 22546 ts_tablet_manager.cc:1939] T dee7242612f44efe963f8059e86b15ab P 3e161600abfa422d94b9de50b3720bf5: Deleting consensus metadata
I20260925 09:18:34.178105 22042 catalog_manager.cc:4841] TS 3e161600abfa422d94b9de50b3720bf5 (127.3.19.67:39491): tablet dee7242612f44efe963f8059e86b15ab (table test-table [id=7d78821ffd3a493cb13d674920da1f02]) successfully deleted
I20260925 09:18:34.263954 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 22114
I20260925 09:18:34.264016 22042 catalog_manager.cc:4841] TS 2b584d594d2f46e8b6ee8ac4022b895e (127.3.19.65:36387): tablet dee7242612f44efe963f8059e86b15ab (table test-table [id=7d78821ffd3a493cb13d674920da1f02]) successfully deleted
I20260925 09:18:34.282054 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 22245
I20260925 09:18:34.288466 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:34.282761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.3.19.66:33095)
0925 09:18:34.282915 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:34.282920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:34.282937 (+ 17us) server_negotiation.cc:407] Connection header received
0925 09:18:34.283057 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:34.283061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:34.283150 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:34.283237 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:34.283585 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.284319 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.285686 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.286067 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.286163 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:34.287237 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:34.287245 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:34.287259 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:34.287824 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:34.287887 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:34.287908 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:34.288008 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:34.288130 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 09:18:34.288213 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 09:18:34.289168 22367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:34.376614 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 22245
I20260925 09:18:34.394390 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 22375
I20260925 09:18:34.400852 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:34.395189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.3.19.67:39491)
0925 09:18:34.395362 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:34.395366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:34.395387 (+ 21us) server_negotiation.cc:407] Connection header received
0925 09:18:34.395492 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:34.395497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:34.395602 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:34.395697 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:34.396118 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.396867 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.398159 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:34.398515 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:34.398607 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:34.399670 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:34.399678 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:34.399692 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:34.400256 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:34.400305 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:34.400323 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:34.400424 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:34.400552 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 09:18:34.400635 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 09:18:34.401487 22497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:34.486816 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 22375
I20260925 09:18:34.503765 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 22024
I20260925 09:18:34.504887 22085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:34.612774 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 22024
2026-09-25T09:18:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2745 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:18:34 dist-test-slave-l6qh krb5kdc[22554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:18:34 dist-test-slave-l6qh krb5kdc[22554](info): set up 2 sockets
Sep 25 09:18:34 dist-test-slave-l6qh krb5kdc[22554](info): commencing operation
krb5kdc: starting...
W20260925 09:18:35.710480 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:18:35 dist-test-slave-l6qh krb5kdc[22554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:18:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:18:35.953953 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33533
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:33533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:36.276122 22567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:36.276484 22567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:36.276686 22567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:36.283813 22567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:36.283969 22567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:36.284061 22567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:36.284147 22567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:36.284230 22567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:36.293753 22567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:33533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:36.295046 22567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:36.296370 22567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:36.307984 22572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:36.308033 22575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:36.307941 22573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:36.308845 22567 server_base.cc:1034] running on GCE node
I20260925 09:18:36.309662 22567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:36.310969 22567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:36.312191 22567 hybrid_clock.cc:648] HybridClock initialized: now 1790327916312140 us; error 62 us; skew 500 ppm
I20260925 09:18:36.312538 22567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:36 dist-test-slave-l6qh krb5kdc[22554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327916, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:36.318807 22567 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:18:36.320246 22567 webserver.cc:460] Webserver started at http://127.3.19.126:44683/ using document root <none> and password file <none>
I20260925 09:18:36.320923 22567 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:36.321091 22567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:36.321460 22567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:36.324342 22567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "c153301f61f84f1b9933762f61be9174"
format_stamp: "Formatted at 2026-09-25 09:18:36 on dist-test-slave-l6qh"
server_key: "25f31dff0cb47c10a6e9b71c2457dd55"
server_key_iv: "b402327f6539def6014a8ef2f5db3507"
server_key_version: "encryptionkey@0"
I20260925 09:18:36.325052 22567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "c153301f61f84f1b9933762f61be9174"
format_stamp: "Formatted at 2026-09-25 09:18:36 on dist-test-slave-l6qh"
server_key: "25f31dff0cb47c10a6e9b71c2457dd55"
server_key_iv: "b402327f6539def6014a8ef2f5db3507"
server_key_version: "encryptionkey@0"
I20260925 09:18:36.329960 22567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 09:18:36.333071 22582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:36.334242 22567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 09:18:36.334436 22567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "c153301f61f84f1b9933762f61be9174"
format_stamp: "Formatted at 2026-09-25 09:18:36 on dist-test-slave-l6qh"
server_key: "25f31dff0cb47c10a6e9b71c2457dd55"
server_key_iv: "b402327f6539def6014a8ef2f5db3507"
server_key_version: "encryptionkey@0"
I20260925 09:18:36.334619 22567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:36.347796 22567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:36.351468 22567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:36.351677 22567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:36.368713 22567 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:33533
I20260925 09:18:36.368739 22633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:33533 every 8 connection(s)
I20260925 09:18:36.370199 22567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:36.373719 22634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:36.376691 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 22567
I20260925 09:18:36.376873 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:36.377364 3149 external_mini_cluster.cc:1428] Setting key 0fd937d5269e563a8cc39d360e7df77f
I20260925 09:18:36.386330 22634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174: Bootstrap starting.
I20260925 09:18:36.389814 22634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:36.390756 22634 log.cc:824] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174: Log is configured to *not* fsync() on all Append() calls
Sep 25 09:18:36 dist-test-slave-l6qh krb5kdc[22554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:18:36.393633 22634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174: No bootstrap required, opened a new log
I20260925 09:18:36.400152 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:36.379055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.3.19.126:33533)
0925 09:18:36.379853 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:36.379868 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:36.379916 (+ 48us) server_negotiation.cc:407] Connection header received
0925 09:18:36.382195 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:36.382245 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:36.382727 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:36.383139 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:36.384474 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:36.386467 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:36.387754 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:36.388775 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:36.388952 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:36.392851 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:36.392863 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:36.392880 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:36.392921 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:36.396178 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:36.396878 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:36.396888 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:36.396912 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:36.397069 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:36.397507 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:36.397516 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:36.397525 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:36.398016 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:36.398362 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:36.398779 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 09:18:36.399187 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":341,"threads_started":1}
I20260925 09:18:36.401520 22634 raft_consensus.cc:348] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c153301f61f84f1b9933762f61be9174" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33533 } }
I20260925 09:18:36.401974 22634 raft_consensus.cc:374] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:36.402076 22634 raft_consensus.cc:729] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c153301f61f84f1b9933762f61be9174, State: Initialized, Role: FOLLOWER
I20260925 09:18:36.402669 22634 consensus_queue.cc:260] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c153301f61f84f1b9933762f61be9174" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33533 } }
I20260925 09:18:36.402973 22634 raft_consensus.cc:388] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:36.403115 22634 raft_consensus.cc:482] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:36.403263 22634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:36.405547 22634 raft_consensus.cc:504] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c153301f61f84f1b9933762f61be9174" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33533 } }
I20260925 09:18:36.406152 22634 leader_election.cc:302] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c153301f61f84f1b9933762f61be9174; no voters:
I20260925 09:18:36.406741 22634 leader_election.cc:288] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:36.406883 22639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:36.407617 22639 raft_consensus.cc:686] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [term 1 LEADER]: Becoming Leader. State: Replica: c153301f61f84f1b9933762f61be9174, State: Running, Role: LEADER
I20260925 09:18:36.408154 22639 consensus_queue.cc:237] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c153301f61f84f1b9933762f61be9174" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33533 } }
I20260925 09:18:36.408780 22634 sys_catalog.cc:564] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:36.410995 22640 sys_catalog.cc:455] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c153301f61f84f1b9933762f61be9174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c153301f61f84f1b9933762f61be9174" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33533 } } }
I20260925 09:18:36.411443 22640 sys_catalog.cc:458] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:36.413798 22641 sys_catalog.cc:455] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c153301f61f84f1b9933762f61be9174. Latest consensus state: current_term: 1 leader_uuid: "c153301f61f84f1b9933762f61be9174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c153301f61f84f1b9933762f61be9174" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33533 } } }
I20260925 09:18:36.414211 22641 sys_catalog.cc:458] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:36.414974 22648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:36.419416 22648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:36.428556 22648 catalog_manager.cc:1347] Generated new cluster ID: d394727f08fa4f35bdd1afbe64e5a442
I20260925 09:18:36.428647 22648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:36.451983 22648 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:36.453299 22648 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:36.467067 22648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174: Generated new TSK 0
I20260925 09:18:36.467821 22648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
Entry for principal kudu/127.3.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab.
I20260925 09:18:36.528223 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33533
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:36.856617 22660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:36.856880 22660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:36.856972 22660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:36.864300 22660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:36.864411 22660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:36.864526 22660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:18:36.874286 22660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.65.keytab
--principal=kudu/127.3.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:36.875456 22660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:36.876693 22660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:36.888629 22665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:36.889389 22666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:36.890856 22668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:36.891233 22660 server_base.cc:1034] running on GCE node
I20260925 09:18:36.891810 22660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:36.892764 22660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:36.893985 22660 hybrid_clock.cc:648] HybridClock initialized: now 1790327916893940 us; error 37 us; skew 500 ppm
I20260925 09:18:36.894351 22660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:36 dist-test-slave-l6qh krb5kdc[22554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327916, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:36.897560 22660 init.cc:377] Logged in from keytab as kudu/127.3.19.65@KRBTEST.COM (short username kudu)
I20260925 09:18:36.898931 22660 webserver.cc:460] Webserver started at http://127.3.19.65:36941/ using document root <none> and password file <none>
I20260925 09:18:36.899691 22660 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:36.899838 22660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:36.900151 22660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:36.903129 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "14efd9db675a46868d31f256c3f236ea"
format_stamp: "Formatted at 2026-09-25 09:18:36 on dist-test-slave-l6qh"
server_key: "6236dcd28111338a700f90f425a97e13"
server_key_iv: "da0107d79b67cdc50af0dde25635b332"
server_key_version: "encryptionkey@0"
I20260925 09:18:36.903882 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "14efd9db675a46868d31f256c3f236ea"
format_stamp: "Formatted at 2026-09-25 09:18:36 on dist-test-slave-l6qh"
server_key: "6236dcd28111338a700f90f425a97e13"
server_key_iv: "da0107d79b67cdc50af0dde25635b332"
server_key_version: "encryptionkey@0"
I20260925 09:18:36.909024 22660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 09:18:36.912549 22675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:36.914045 22660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:18:36.914209 22660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "14efd9db675a46868d31f256c3f236ea"
format_stamp: "Formatted at 2026-09-25 09:18:36 on dist-test-slave-l6qh"
server_key: "6236dcd28111338a700f90f425a97e13"
server_key_iv: "da0107d79b67cdc50af0dde25635b332"
server_key_version: "encryptionkey@0"
I20260925 09:18:36.914407 22660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:36.932476 22660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:36.936545 22660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:36.936795 22660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:36.937839 22660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:36.939319 22660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:36.939411 22660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:36.939509 22660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:36.939554 22660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:36.973731 22660 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:34435
I20260925 09:18:36.973798 22787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:34435 every 8 connection(s)
I20260925 09:18:36.975060 22660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:18:36.982188 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 22660
I20260925 09:18:36.982388 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:18:36.982843 3149 external_mini_cluster.cc:1428] Setting key 481cf6f8ab3b19a05a25bade0f835439
Sep 25 09:18:36 dist-test-slave-l6qh krb5kdc[22554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327916, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.65@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:18:36.995522 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:36.977622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:57799 (local address 127.3.19.126:33533)
0925 09:18:36.978054 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:36.978059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:36.978917 (+ 858us) server_negotiation.cc:407] Connection header received
0925 09:18:36.980761 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:36.980767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:36.980856 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:36.980975 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:36.982633 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:36.983496 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:36.985610 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:36.986013 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:36.986165 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:36.989674 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:36.989679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:36.989682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:36.989713 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:36.992164 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:36.992880 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:36.992887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:36.992895 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:36.993007 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:36.993495 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:36.993502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:36.993509 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:36.993892 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:36.994048 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:36.994773 (+ 725us) server_negotiation.cc:299] Negotiation successful
0925 09:18:36.995028 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":242,"threads_started":1}
I20260925 09:18:36.996157 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:36.978054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33533 (local address 127.3.19.65:57799)
0925 09:18:36.978763 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:36.978805 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:36.980499 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:36.981215 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:36.981258 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:36.981800 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:36.982460 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:36.982488 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:36.983713 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:36.983716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:36.985469 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:36.985481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:36.986244 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:36.986255 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:36.986492 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:36.987225 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:18:36.987251 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:36.989454 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:36.992307 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:36.992316 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:36.992332 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:36.992752 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:36.993115 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:36.993121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:36.993124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:36.993378 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:36.994035 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:36.994040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:36.994374 (+ 334us) client_negotiation.cc:769] Sending connection context
0925 09:18:36.994615 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 09:18:36.995001 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":288,"threads_started":1}
I20260925 09:18:36.997833 22788 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33533
I20260925 09:18:36.998168 22788 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:36.999061 22788 heartbeater.cc:511] Master 127.3.19.126:33533 requested a full tablet report, sending...
I20260925 09:18:37.001503 22599 ts_manager.cc:194] Registered new tserver with Master: 14efd9db675a46868d31f256c3f236ea (127.3.19.65:34435)
I20260925 09:18:37.003055 22599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.65@KRBTEST.COM'} at 127.3.19.65:57799
WARNING: no policy specified for kudu/127.3.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
Entry for principal kudu/127.3.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab.
I20260925 09:18:37.044060 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33533
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:37.374168 22794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:37.374469 22794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:37.374632 22794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:37.382593 22794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:37.382745 22794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:37.382918 22794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:18:37.392683 22794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.66.keytab
--principal=kudu/127.3.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:37.394013 22794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:37.395296 22794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:37.408829 22800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:37.408804 22799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:37.408761 22802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:37.409709 22794 server_base.cc:1034] running on GCE node
I20260925 09:18:37.410431 22794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:37.411434 22794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:37.412642 22794 hybrid_clock.cc:648] HybridClock initialized: now 1790327917412577 us; error 53 us; skew 500 ppm
I20260925 09:18:37.412971 22794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:37 dist-test-slave-l6qh krb5kdc[22554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327917, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:37.416540 22794 init.cc:377] Logged in from keytab as kudu/127.3.19.66@KRBTEST.COM (short username kudu)
I20260925 09:18:37.418262 22794 webserver.cc:460] Webserver started at http://127.3.19.66:46273/ using document root <none> and password file <none>
I20260925 09:18:37.418987 22794 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:37.419144 22794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:37.419489 22794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:37.422683 22794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "852590d141d1412e839aa5dee87aa986"
format_stamp: "Formatted at 2026-09-25 09:18:37 on dist-test-slave-l6qh"
server_key: "106d8057f678a21dae46dfdee869d729"
server_key_iv: "a09211f2207c9fcc00d393f23cad1356"
server_key_version: "encryptionkey@0"
I20260925 09:18:37.423575 22794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "852590d141d1412e839aa5dee87aa986"
format_stamp: "Formatted at 2026-09-25 09:18:37 on dist-test-slave-l6qh"
server_key: "106d8057f678a21dae46dfdee869d729"
server_key_iv: "a09211f2207c9fcc00d393f23cad1356"
server_key_version: "encryptionkey@0"
I20260925 09:18:37.429040 22794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 09:18:37.432700 22809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:37.434087 22794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:18:37.434268 22794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "852590d141d1412e839aa5dee87aa986"
format_stamp: "Formatted at 2026-09-25 09:18:37 on dist-test-slave-l6qh"
server_key: "106d8057f678a21dae46dfdee869d729"
server_key_iv: "a09211f2207c9fcc00d393f23cad1356"
server_key_version: "encryptionkey@0"
I20260925 09:18:37.434484 22794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:37.462291 22794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:37.466670 22794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:37.466954 22794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:37.467958 22794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:37.469348 22794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:37.469446 22794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:37.469591 22794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:37.469663 22794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:37.502144 22794 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:33091
I20260925 09:18:37.502192 22921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:33091 every 8 connection(s)
I20260925 09:18:37.503396 22794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:18:37.509819 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 22794
I20260925 09:18:37.510066 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:18:37.510555 3149 external_mini_cluster.cc:1428] Setting key 3a47aa7ddc528837846cf5f4c243fd03
Sep 25 09:18:37 dist-test-slave-l6qh krb5kdc[22554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327917, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.66@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:18:37.526500 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:37.506236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:54755 (local address 127.3.19.126:33533)
0925 09:18:37.506678 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:37.506682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:37.507643 (+ 961us) server_negotiation.cc:407] Connection header received
0925 09:18:37.509491 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:37.509496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:37.509569 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:37.509714 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:37.511602 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:37.512717 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:37.515360 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:37.515785 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:37.515877 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:37.520190 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:37.520195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:37.520198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:37.520229 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:37.523134 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:37.523854 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:37.523859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:37.523864 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:37.524022 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:37.524459 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:37.524466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:37.524472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:37.524839 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:37.525000 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:37.525733 (+ 733us) server_negotiation.cc:299] Negotiation successful
0925 09:18:37.525979 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":215,"threads_started":1}
I20260925 09:18:37.527132 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:37.506816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33533 (local address 127.3.19.66:54755)
0925 09:18:37.507491 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:37.507530 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:37.509221 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:37.509990 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:37.510002 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:37.510701 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:37.511390 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:37.511415 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:37.512891 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:37.512897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:37.515197 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:37.515218 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:37.516021 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:37.516031 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:37.516383 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:37.517165 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:18:37.517211 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:37.519943 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:37.523278 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:37.523284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:37.523296 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:37.523720 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:37.524127 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:37.524131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:37.524133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:37.524315 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:37.524963 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:37.524969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:37.525345 (+ 376us) client_negotiation.cc:769] Sending connection context
0925 09:18:37.525599 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 09:18:37.525976 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":287,"threads_started":1}
I20260925 09:18:37.528621 22922 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33533
I20260925 09:18:37.528985 22922 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:37.529840 22922 heartbeater.cc:511] Master 127.3.19.126:33533 requested a full tablet report, sending...
I20260925 09:18:37.531493 22599 ts_manager.cc:194] Registered new tserver with Master: 852590d141d1412e839aa5dee87aa986 (127.3.19.66:33091)
I20260925 09:18:37.532313 22599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.66@KRBTEST.COM'} at 127.3.19.66:54755
WARNING: no policy specified for kudu/127.3.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
Entry for principal kudu/127.3.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab.
I20260925 09:18:37.572456 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33533
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:37.899188 22928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:37.899461 22928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:37.899667 22928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:37.906508 22928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:37.906668 22928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:37.906850 22928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:18:37.916272 22928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.67.keytab
--principal=kudu/127.3.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:33533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:37.917538 22928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:37.918887 22928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:37.931808 22933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:37.931752 22936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:37.931830 22934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:37.934566 22928 server_base.cc:1034] running on GCE node
I20260925 09:18:37.935163 22928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:37.936085 22928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:37.937350 22928 hybrid_clock.cc:648] HybridClock initialized: now 1790327917937298 us; error 58 us; skew 500 ppm
I20260925 09:18:37.937759 22928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:37 dist-test-slave-l6qh krb5kdc[22554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327917, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:37.940961 22928 init.cc:377] Logged in from keytab as kudu/127.3.19.67@KRBTEST.COM (short username kudu)
I20260925 09:18:37.942363 22928 webserver.cc:460] Webserver started at http://127.3.19.67:45275/ using document root <none> and password file <none>
I20260925 09:18:37.943040 22928 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:37.943146 22928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:37.943397 22928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:37.946259 22928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "8edd37d790204dd28ab43b21264a4381"
format_stamp: "Formatted at 2026-09-25 09:18:37 on dist-test-slave-l6qh"
server_key: "471222b02cf496cf0f472bcf8c03aa85"
server_key_iv: "b557b310b5ad458198283b54619c4ba2"
server_key_version: "encryptionkey@0"
I20260925 09:18:37.946952 22928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "8edd37d790204dd28ab43b21264a4381"
format_stamp: "Formatted at 2026-09-25 09:18:37 on dist-test-slave-l6qh"
server_key: "471222b02cf496cf0f472bcf8c03aa85"
server_key_iv: "b557b310b5ad458198283b54619c4ba2"
server_key_version: "encryptionkey@0"
I20260925 09:18:37.951660 22928 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 09:18:37.954828 22943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:37.956094 22928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:18:37.956271 22928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "8edd37d790204dd28ab43b21264a4381"
format_stamp: "Formatted at 2026-09-25 09:18:37 on dist-test-slave-l6qh"
server_key: "471222b02cf496cf0f472bcf8c03aa85"
server_key_iv: "b557b310b5ad458198283b54619c4ba2"
server_key_version: "encryptionkey@0"
I20260925 09:18:37.956458 22928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:37.980254 22928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:37.983987 22928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:37.984256 22928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:37.985146 22928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:37.986449 22928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:37.986549 22928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:37.986693 22928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:37.986766 22928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:38.006809 22788 heartbeater.cc:503] Master 127.3.19.126:33533 was elected leader, sending a full tablet report...
I20260925 09:18:38.020699 22928 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:42733
I20260925 09:18:38.020742 23055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:42733 every 8 connection(s)
I20260925 09:18:38.021981 22928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:18:38.026654 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 22928
I20260925 09:18:38.026846 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:18:38.027426 3149 external_mini_cluster.cc:1428] Setting key 6d38089a06debce5256d01e5a62980af
Sep 25 09:18:38 dist-test-slave-l6qh krb5kdc[22554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327917, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.67@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:18:38.042371 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.024534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:59891 (local address 127.3.19.126:33533)
0925 09:18:38.024718 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:38.024723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:38.025917 (+ 1194us) server_negotiation.cc:407] Connection header received
0925 09:18:38.028380 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:38.028385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:38.028483 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:38.028587 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:38.030484 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.031180 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.032607 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.032983 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.033072 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.036507 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:38.036511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:38.036514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:38.036542 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:38.039256 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:38.039946 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:38.039950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:38.039952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:38.040065 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:38.040433 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:38.040436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:38.040438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:38.040698 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:38.040974 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:38.041792 (+ 818us) server_negotiation.cc:299] Negotiation successful
0925 09:18:38.041980 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 09:18:38.043493 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.024970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:33533 (local address 127.3.19.67:59891)
0925 09:18:38.025750 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:38.025790 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:38.028026 (+ 2236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:38.028865 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:38.028876 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:38.029634 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:38.030320 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:38.030344 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.031296 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.031300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:38.032489 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:38.032501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.033167 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.033171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:38.033433 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.034117 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:18:38.034138 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:38.036322 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:38.039388 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:38.039394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:38.039405 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:38.039830 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:38.040162 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:18:38.040165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:18:38.040167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:18:38.040336 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:18:38.040952 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:38.040959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:38.041384 (+ 425us) client_negotiation.cc:769] Sending connection context
0925 09:18:38.041639 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 09:18:38.042191 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":346,"threads_started":1}
I20260925 09:18:38.044723 23056 heartbeater.cc:347] Connected to a master server at 127.3.19.126:33533
I20260925 09:18:38.044991 23056 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:38.045711 23056 heartbeater.cc:511] Master 127.3.19.126:33533 requested a full tablet report, sending...
I20260925 09:18:38.047060 22599 ts_manager.cc:194] Registered new tserver with Master: 8edd37d790204dd28ab43b21264a4381 (127.3.19.67:42733)
I20260925 09:18:38.047837 22599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.19.67@KRBTEST.COM'} at 127.3.19.67:59891
I20260925 09:18:38.055137 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:18:38.070655 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.059180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59716 (local address 127.3.19.126:33533)
0925 09:18:38.059358 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:38.059363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:38.059903 (+ 540us) server_negotiation.cc:407] Connection header received
0925 09:18:38.060025 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:38.060029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:38.060105 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:38.060183 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:38.061722 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.062392 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.063524 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.063887 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.063967 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.066208 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:38.066212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:38.066215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:38.066242 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:38.068454 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:38.068998 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:38.069001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:38.069003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:38.069119 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:38.069470 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:38.069473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:38.069475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:38.069749 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:38.069950 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:38.070251 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 09:18:38.070392 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 09:18:38.074111 22599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:18:38.080770 22599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:18:38.116006 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.092771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.3.19.67:42733)
0925 09:18:38.099861 (+ 7090us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:38.099874 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:38.105252 (+ 5378us) server_negotiation.cc:407] Connection header received
0925 09:18:38.107045 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:38.107054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:38.107339 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:38.107583 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:38.108950 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.110340 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.113486 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.114809 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.114920 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.115099 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:38.115485 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 09:18:38.115649 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6828,"thread_start_us":6632,"threads_started":1}
I20260925 09:18:38.116163 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.092934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:33091 (local address 127.0.0.1:37146)
0925 09:18:38.106515 (+ 13581us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:38.106538 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:38.106733 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:38.107861 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:38.107870 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:38.107923 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:38.108932 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:38.108948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.110908 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.110913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:38.113075 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:38.113086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.114912 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.114921 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:38.115218 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.115285 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 09:18:38.115409 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 09:18:38.115675 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13419,"mutex_wait_us":124,"spinlock_wait_cycles":3840,"thread_start_us":13289,"threads_started":1}
I20260925 09:18:38.117085 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.092393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:42733 (local address 127.0.0.1:44954)
0925 09:18:38.105022 (+ 12629us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:38.105075 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:38.106813 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:38.107863 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:38.107870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:38.107928 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:38.108705 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:38.108720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.110466 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.110472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:38.113329 (+ 2857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:38.113342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.114985 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.114990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:38.115271 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.115295 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 09:18:38.115422 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 09:18:38.115680 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12315,"mutex_wait_us":270,"spinlock_wait_cycles":20736,"thread_start_us":14162,"threads_started":1}
I20260925 09:18:38.117127 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.093506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.3.19.66:33091)
0925 09:18:38.101345 (+ 7839us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:38.101357 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:38.106970 (+ 5613us) server_negotiation.cc:407] Connection header received
0925 09:18:38.107072 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:38.107082 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:38.107310 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:38.107999 (+ 689us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:38.109135 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.110736 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.113254 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.114701 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.114862 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.115065 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:38.116624 (+ 1559us) server_negotiation.cc:299] Negotiation successful
0925 09:18:38.116799 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7635,"thread_start_us":7506,"threads_started":1}
I20260925 09:18:38.117712 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.093274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34435 (local address 127.0.0.1:46910)
0925 09:18:38.106829 (+ 13555us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:38.106852 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:38.107000 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:38.107875 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:38.107880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:38.107940 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:38.108531 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:38.108549 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.110585 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.110590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:38.113328 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:38.113342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.116873 (+ 3531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.116879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:38.117057 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.117101 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 09:18:38.117219 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 09:18:38.117351 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13441,"mutex_wait_us":114,"spinlock_wait_cycles":29056,"thread_start_us":11267,"threads_started":1}
I20260925 09:18:38.118064 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.093660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.3.19.65:34435)
0925 09:18:38.105753 (+ 12093us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:38.105761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:38.107034 (+ 1273us) server_negotiation.cc:407] Connection header received
0925 09:18:38.107158 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:38.107164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:38.107373 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:38.107582 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:38.108791 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.110457 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.115152 (+ 4695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.116679 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.116828 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.117061 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:38.117429 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 09:18:38.117592 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11924,"thread_start_us":12034,"threads_started":1}
I20260925 09:18:38.121531 22991 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec8ad6f7bc54e38b8b0504dfa60e346 (DEFAULT_TABLE table=test-table [id=ae6062233cc54ea3be6202e122ae1922]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:38.121562 22857 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec8ad6f7bc54e38b8b0504dfa60e346 (DEFAULT_TABLE table=test-table [id=ae6062233cc54ea3be6202e122ae1922]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:38.122350 22723 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec8ad6f7bc54e38b8b0504dfa60e346 (DEFAULT_TABLE table=test-table [id=ae6062233cc54ea3be6202e122ae1922]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:38.122938 22991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec8ad6f7bc54e38b8b0504dfa60e346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:38.122938 22857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec8ad6f7bc54e38b8b0504dfa60e346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:38.123456 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec8ad6f7bc54e38b8b0504dfa60e346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:38.135537 23074 tablet_bootstrap.cc:492] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea: Bootstrap starting.
I20260925 09:18:38.138253 23075 tablet_bootstrap.cc:492] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986: Bootstrap starting.
I20260925 09:18:38.138443 23076 tablet_bootstrap.cc:492] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381: Bootstrap starting.
I20260925 09:18:38.139493 23074 tablet_bootstrap.cc:654] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:38.140790 23074 log.cc:824] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:38.141841 23075 tablet_bootstrap.cc:654] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:38.143074 23075 log.cc:824] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:38.143951 23074 tablet_bootstrap.cc:492] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea: No bootstrap required, opened a new log
I20260925 09:18:38.144223 23074 ts_tablet_manager.cc:1397] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 09:18:38.144416 23076 tablet_bootstrap.cc:654] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:38.145395 23076 log.cc:824] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:38.146250 23075 tablet_bootstrap.cc:492] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986: No bootstrap required, opened a new log
I20260925 09:18:38.146502 23075 ts_tablet_manager.cc:1397] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260925 09:18:38.147689 23076 tablet_bootstrap.cc:492] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381: No bootstrap required, opened a new log
I20260925 09:18:38.147921 23076 ts_tablet_manager.cc:1397] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20260925 09:18:38.154873 23076 raft_consensus.cc:348] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8edd37d790204dd28ab43b21264a4381" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 42733 } } peers { permanent_uuid: "852590d141d1412e839aa5dee87aa986" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33091 } } peers { permanent_uuid: "14efd9db675a46868d31f256c3f236ea" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34435 } }
I20260925 09:18:38.154868 23074 raft_consensus.cc:348] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8edd37d790204dd28ab43b21264a4381" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 42733 } } peers { permanent_uuid: "852590d141d1412e839aa5dee87aa986" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33091 } } peers { permanent_uuid: "14efd9db675a46868d31f256c3f236ea" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34435 } }
I20260925 09:18:38.155222 23076 raft_consensus.cc:374] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:38.155303 23074 raft_consensus.cc:374] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:38.155330 23076 raft_consensus.cc:729] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8edd37d790204dd28ab43b21264a4381, State: Initialized, Role: FOLLOWER
I20260925 09:18:38.155398 23074 raft_consensus.cc:729] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14efd9db675a46868d31f256c3f236ea, State: Initialized, Role: FOLLOWER
I20260925 09:18:38.156065 23076 consensus_queue.cc:260] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8edd37d790204dd28ab43b21264a4381" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 42733 } } peers { permanent_uuid: "852590d141d1412e839aa5dee87aa986" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33091 } } peers { permanent_uuid: "14efd9db675a46868d31f256c3f236ea" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34435 } }
I20260925 09:18:38.156020 23074 consensus_queue.cc:260] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8edd37d790204dd28ab43b21264a4381" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 42733 } } peers { permanent_uuid: "852590d141d1412e839aa5dee87aa986" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33091 } } peers { permanent_uuid: "14efd9db675a46868d31f256c3f236ea" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34435 } }
I20260925 09:18:38.156826 23075 raft_consensus.cc:348] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8edd37d790204dd28ab43b21264a4381" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 42733 } } peers { permanent_uuid: "852590d141d1412e839aa5dee87aa986" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33091 } } peers { permanent_uuid: "14efd9db675a46868d31f256c3f236ea" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34435 } }
I20260925 09:18:38.157307 23075 raft_consensus.cc:374] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:38.157449 23075 raft_consensus.cc:729] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852590d141d1412e839aa5dee87aa986, State: Initialized, Role: FOLLOWER
I20260925 09:18:38.157816 23074 ts_tablet_manager.cc:1428] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260925 09:18:38.158025 23075 consensus_queue.cc:260] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8edd37d790204dd28ab43b21264a4381" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 42733 } } peers { permanent_uuid: "852590d141d1412e839aa5dee87aa986" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33091 } } peers { permanent_uuid: "14efd9db675a46868d31f256c3f236ea" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34435 } }
I20260925 09:18:38.158739 23056 heartbeater.cc:503] Master 127.3.19.126:33533 was elected leader, sending a full tablet report...
I20260925 09:18:38.159466 23076 ts_tablet_manager.cc:1428] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381: Time spent starting tablet: real 0.011s user 0.012s sys 0.001s
I20260925 09:18:38.162416 22922 heartbeater.cc:503] Master 127.3.19.126:33533 was elected leader, sending a full tablet report...
I20260925 09:18:38.163551 23075 ts_tablet_manager.cc:1428] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986: Time spent starting tablet: real 0.017s user 0.004s sys 0.011s
W20260925 09:18:38.227334 22789 tablet.cc:2369] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:18:38.255643 22923 tablet.cc:2369] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:18:38.273705 23057 tablet.cc:2369] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:18:38.332877 23080 raft_consensus.cc:482] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:18:38.333182 23080 raft_consensus.cc:504] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8edd37d790204dd28ab43b21264a4381" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 42733 } } peers { permanent_uuid: "852590d141d1412e839aa5dee87aa986" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33091 } } peers { permanent_uuid: "14efd9db675a46868d31f256c3f236ea" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34435 } }
I20260925 09:18:38.334910 23080 leader_election.cc:288] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8edd37d790204dd28ab43b21264a4381 (127.3.19.67:42733), 852590d141d1412e839aa5dee87aa986 (127.3.19.66:33091)
I20260925 09:18:38.342120 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.335227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:56789 (local address 127.3.19.67:42733)
0925 09:18:38.335431 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:38.335439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:38.336110 (+ 671us) server_negotiation.cc:407] Connection header received
0925 09:18:38.336362 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:38.336366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:38.336423 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:38.336516 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:38.337149 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.338065 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.340387 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.341135 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.341299 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.341405 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:38.341730 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 09:18:38.341856 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 09:18:38.342120 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.335399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:59583 (local address 127.3.19.66:33091)
0925 09:18:38.335662 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:38.335671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:38.336327 (+ 656us) server_negotiation.cc:407] Connection header received
0925 09:18:38.336520 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:38.336524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:38.336581 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:38.336682 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:38.337397 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.338249 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.340383 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.341102 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.341282 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.341398 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:38.341743 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 09:18:38.341864 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 09:18:38.342296 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.335331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:33091 (local address 127.3.19.65:59583)
0925 09:18:38.336220 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:38.336233 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:38.336386 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:38.336697 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:38.336701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:38.336738 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:38.337189 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:38.337201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.338373 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.338379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:38.340232 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:38.340243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.341428 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.341435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:38.341558 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.341598 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 09:18:38.341708 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 09:18:38.341842 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":7,"thread_start_us":558,"threads_started":1}
I20260925 09:18:38.342337 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.335201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:42733 (local address 127.3.19.65:56789)
0925 09:18:38.335990 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:38.336005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:38.336209 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:38.336543 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:38.336547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:38.336577 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:38.336983 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:38.336995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.338186 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.338193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:38.340241 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:38.340254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.341382 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.341390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:38.341549 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.341602 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 09:18:38.341704 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 09:18:38.341857 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":60,"spinlock_wait_cycles":17280,"thread_start_us":455,"threads_started":1}
I20260925 09:18:38.343331 23011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec8ad6f7bc54e38b8b0504dfa60e346" candidate_uuid: "14efd9db675a46868d31f256c3f236ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edd37d790204dd28ab43b21264a4381" is_pre_election: true
I20260925 09:18:38.343343 22877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec8ad6f7bc54e38b8b0504dfa60e346" candidate_uuid: "14efd9db675a46868d31f256c3f236ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852590d141d1412e839aa5dee87aa986" is_pre_election: true
I20260925 09:18:38.343817 22877 raft_consensus.cc:2454] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14efd9db675a46868d31f256c3f236ea in term 0.
I20260925 09:18:38.343817 23011 raft_consensus.cc:2454] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14efd9db675a46868d31f256c3f236ea in term 0.
I20260925 09:18:38.344436 22676 leader_election.cc:302] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14efd9db675a46868d31f256c3f236ea, 852590d141d1412e839aa5dee87aa986, 8edd37d790204dd28ab43b21264a4381; no voters:
I20260925 09:18:38.344805 23080 raft_consensus.cc:2781] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:18:38.344959 23080 raft_consensus.cc:482] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:18:38.345067 23080 raft_consensus.cc:3037] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:38.347242 23080 raft_consensus.cc:504] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8edd37d790204dd28ab43b21264a4381" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 42733 } } peers { permanent_uuid: "852590d141d1412e839aa5dee87aa986" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33091 } } peers { permanent_uuid: "14efd9db675a46868d31f256c3f236ea" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34435 } }
I20260925 09:18:38.347899 23080 leader_election.cc:288] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [CANDIDATE]: Term 1 election: Requested vote from peers 8edd37d790204dd28ab43b21264a4381 (127.3.19.67:42733), 852590d141d1412e839aa5dee87aa986 (127.3.19.66:33091)
I20260925 09:18:38.348289 23011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec8ad6f7bc54e38b8b0504dfa60e346" candidate_uuid: "14efd9db675a46868d31f256c3f236ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edd37d790204dd28ab43b21264a4381"
I20260925 09:18:38.348395 22877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec8ad6f7bc54e38b8b0504dfa60e346" candidate_uuid: "14efd9db675a46868d31f256c3f236ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852590d141d1412e839aa5dee87aa986"
I20260925 09:18:38.348500 23011 raft_consensus.cc:3037] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:38.348578 22877 raft_consensus.cc:3037] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:38.351637 23011 raft_consensus.cc:2454] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14efd9db675a46868d31f256c3f236ea in term 1.
I20260925 09:18:38.351635 22877 raft_consensus.cc:2454] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14efd9db675a46868d31f256c3f236ea in term 1.
I20260925 09:18:38.352020 22676 leader_election.cc:302] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14efd9db675a46868d31f256c3f236ea, 852590d141d1412e839aa5dee87aa986; no voters:
I20260925 09:18:38.352325 23080 raft_consensus.cc:2781] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:38.352782 23080 raft_consensus.cc:686] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [term 1 LEADER]: Becoming Leader. State: Replica: 14efd9db675a46868d31f256c3f236ea, State: Running, Role: LEADER
I20260925 09:18:38.353350 23080 consensus_queue.cc:237] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8edd37d790204dd28ab43b21264a4381" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 42733 } } peers { permanent_uuid: "852590d141d1412e839aa5dee87aa986" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33091 } } peers { permanent_uuid: "14efd9db675a46868d31f256c3f236ea" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34435 } }
I20260925 09:18:38.357743 22599 catalog_manager.cc:5495] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 14efd9db675a46868d31f256c3f236ea (127.3.19.65). New cstate: current_term: 1 leader_uuid: "14efd9db675a46868d31f256c3f236ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8edd37d790204dd28ab43b21264a4381" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 42733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852590d141d1412e839aa5dee87aa986" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14efd9db675a46868d31f256c3f236ea" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 34435 } health_report { overall_health: HEALTHY } } }
I20260925 09:18:38.407352 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.402509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.3.19.65:34435)
0925 09:18:38.402709 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:38.402713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:38.402730 (+ 17us) server_negotiation.cc:407] Connection header received
0925 09:18:38.402874 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:38.402877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:38.402952 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:38.403071 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:18:38.403384 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.404066 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.405425 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.405802 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.405891 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.406316 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:18:38.406813 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:18:38.406917 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:38.407001 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 09:18:38.407059 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 09:18:38.415987 22877 raft_consensus.cc:1261] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986 [term 1 FOLLOWER]: Refusing update from remote peer 14efd9db675a46868d31f256c3f236ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:18:38.416095 23011 raft_consensus.cc:1261] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381 [term 1 FOLLOWER]: Refusing update from remote peer 14efd9db675a46868d31f256c3f236ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:18:38.416999 23085 consensus_queue.cc:1035] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "852590d141d1412e839aa5dee87aa986" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 33091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:18:38.417397 23080 consensus_queue.cc:1035] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8edd37d790204dd28ab43b21264a4381" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:18:38.426397 23088 mvcc.cc:205] Tried to move back new op lower bound from 7333183153822019584 to 7333183153578782720. Current Snapshot: MvccSnapshot[applied={T|T < 7333183153822019584}]
I20260925 09:18:38.432116 23089 mvcc.cc:205] Tried to move back new op lower bound from 7333183153822019584 to 7333183153578782720. Current Snapshot: MvccSnapshot[applied={T|T < 7333183153822019584 or (T in {7333183153822019584})}]
I20260925 09:18:38.443779 22599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:18:38.444106 22599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:18:38.447801 22599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c153301f61f84f1b9933762f61be9174: Sending DeleteTablet for 3 replicas of tablet 9ec8ad6f7bc54e38b8b0504dfa60e346
I20260925 09:18:38.448966 22991 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec8ad6f7bc54e38b8b0504dfa60e346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:38 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:44954
I20260925 09:18:38.449047 22857 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec8ad6f7bc54e38b8b0504dfa60e346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:38 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:37146
I20260925 09:18:38.449406 22723 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec8ad6f7bc54e38b8b0504dfa60e346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:38 UTC) from {username='kudu', principal='kudu/127.3.19.126@KRBTEST.COM'} at 127.0.0.1:46910
I20260925 09:18:38.451189 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 22660
I20260925 09:18:38.461252 23099 tablet_replica.cc:330] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986: stopping tablet replica
I20260925 09:18:38.462553 23099 raft_consensus.cc:2229] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:18:38.463940 23099 raft_consensus.cc:2258] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:38.464685 23098 tablet_replica.cc:330] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381: stopping tablet replica
I20260925 09:18:38.465372 23100 tablet_replica.cc:330] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea: stopping tablet replica
I20260925 09:18:38.465847 23098 raft_consensus.cc:2229] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:18:38.466353 23100 raft_consensus.cc:2229] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:18:38.466917 23098 raft_consensus.cc:2258] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:38.467444 23099 ts_tablet_manager.cc:1905] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:38.468276 23100 raft_consensus.cc:2258] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:38.469749 23098 ts_tablet_manager.cc:1905] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 09:18:38 dist-test-slave-l6qh krb5kdc[22554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.65@KRBTEST.COM
I20260925 09:18:38.471515 23100 ts_tablet_manager.cc:1905] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:38.474143 23099 ts_tablet_manager.cc:1918] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:38.474355 23099 log.cc:1196] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/9ec8ad6f7bc54e38b8b0504dfa60e346
I20260925 09:18:38.474928 23099 ts_tablet_manager.cc:1939] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 852590d141d1412e839aa5dee87aa986: Deleting consensus metadata
I20260925 09:18:38.476936 22583 catalog_manager.cc:4841] TS 852590d141d1412e839aa5dee87aa986 (127.3.19.66:33091): tablet 9ec8ad6f7bc54e38b8b0504dfa60e346 (table test-table [id=ae6062233cc54ea3be6202e122ae1922]) successfully deleted
I20260925 09:18:38.478248 23098 ts_tablet_manager.cc:1918] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:38.478470 23098 log.cc:1196] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/9ec8ad6f7bc54e38b8b0504dfa60e346
I20260925 09:18:38.479048 23098 ts_tablet_manager.cc:1939] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 8edd37d790204dd28ab43b21264a4381: Deleting consensus metadata
I20260925 09:18:38.480958 22585 catalog_manager.cc:4841] TS 8edd37d790204dd28ab43b21264a4381 (127.3.19.67:42733): tablet 9ec8ad6f7bc54e38b8b0504dfa60e346 (table test-table [id=ae6062233cc54ea3be6202e122ae1922]) successfully deleted
I20260925 09:18:38.481982 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.452490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.3.19.65:34435)
0925 09:18:38.452679 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:38.452688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:38.458224 (+ 5536us) server_negotiation.cc:407] Connection header received
0925 09:18:38.458443 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:38.458450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:38.458549 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:38.458664 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:38.462781 (+ 4117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.464014 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.466064 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.466835 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.466940 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.472631 (+ 5691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:38.472646 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:38.472664 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:38.472713 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:38.477825 (+ 5112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:38.478934 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:38.478949 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:38.478953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:38.479297 (+ 344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:38.479895 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:38.479901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:38.479904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:38.480345 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:38.480678 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:38.481307 (+ 629us) server_negotiation.cc:299] Negotiation successful
0925 09:18:38.481592 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 09:18:38.483575 22783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:38.483600 23100 ts_tablet_manager.cc:1918] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:38.483810 23100 log.cc:1196] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/9ec8ad6f7bc54e38b8b0504dfa60e346
I20260925 09:18:38.580783 23100 ts_tablet_manager.cc:1939] T 9ec8ad6f7bc54e38b8b0504dfa60e346 P 14efd9db675a46868d31f256c3f236ea: Deleting consensus metadata
I20260925 09:18:38.580873 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 22660
I20260925 09:18:38.599061 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 22794
W20260925 09:18:38.599208 22586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.19.65:34435 (error 108)
I20260925 09:18:38.600915 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.600524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34435 (local address 127.0.0.1:46928)
0925 09:18:38.600682 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:38.600784 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 09:18:38.601130 22586 catalog_manager.cc:4584] TS 14efd9db675a46868d31f256c3f236ea (127.3.19.65:34435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ec8ad6f7bc54e38b8b0504dfa60e346: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34435: connect: Connection refused (error 111)
Sep 25 09:18:38 dist-test-slave-l6qh krb5kdc[22554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.66@KRBTEST.COM
I20260925 09:18:38.615821 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.599953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.3.19.66:33091)
0925 09:18:38.600166 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:38.600172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:38.600293 (+ 121us) server_negotiation.cc:407] Connection header received
0925 09:18:38.600496 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:38.600499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:38.600606 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:38.600721 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:38.602597 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.603725 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.605482 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.605855 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.605995 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.610220 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:38.610242 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:38.610255 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:38.610315 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:38.613005 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:38.613739 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:38.613755 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:38.613757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:38.613876 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:38.614261 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:38.614266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:38.614269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:38.614651 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:38.614950 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:38.615262 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 09:18:38.615471 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 09:18:38.616454 22917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:38.654971 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.654506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34435 (local address 127.0.0.1:46930)
0925 09:18:38.654723 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:38.654830 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 09:18:38.656026 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.655669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34435 (local address 127.0.0.1:46940)
0925 09:18:38.655831 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:38.655907 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 09:18:38.705592 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 22794
I20260925 09:18:38.716684 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.716079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34435 (local address 127.0.0.1:46950)
0925 09:18:38.716330 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:38.716465 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 09:18:38.717870 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.717426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34435 (local address 127.0.0.1:46958)
0925 09:18:38.717598 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:38.717696 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 09:18:38.724336 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 22928
Sep 25 09:18:38 dist-test-slave-l6qh krb5kdc[22554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.67@KRBTEST.COM
I20260925 09:18:38.739068 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.725018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.3.19.67:42733)
0925 09:18:38.725195 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:38.725199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:38.725217 (+ 18us) server_negotiation.cc:407] Connection header received
0925 09:18:38.725341 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:38.725345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:38.725418 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:38.725521 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:38.726824 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.727514 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.728776 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:38.729150 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:38.729320 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:38.733555 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:38.733570 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:38.733586 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:38.733635 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:38.736352 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:38.736973 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:38.736980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:38.736983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:38.737145 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:38.737650 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:38.737654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:38.737656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:38.737990 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:38.738262 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:38.738577 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 09:18:38.738821 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 09:18:38.739763 23051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:38.798538 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.798054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34435 (local address 127.0.0.1:46966)
0925 09:18:38.798279 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:38.798363 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 09:18:38.799592 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:38.799187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:34435 (local address 127.0.0.1:46974)
0925 09:18:38.799409 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:38.799488 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.65:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 09:18:38.826193 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 22928
I20260925 09:18:38.843207 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 22567
I20260925 09:18:38.844254 22629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:38.952636 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 22567
2026-09-25T09:18:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4341 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T09:18:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:39Z Disabled control of system clock
I20260925 09:18:39.031170 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:41271
--webserver_interface=127.3.19.126
--webserver_port=0
--builtin_ntp_servers=127.3.19.84:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:41271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 09:18:39.355659 23110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:39.355926 23110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:39.356006 23110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:39.363175 23110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:39.363274 23110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:39.363309 23110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:39.363363 23110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:39.363405 23110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:39.372046 23110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:41271
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:41271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:39.373323 23110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:39.374940 23110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:39.386976 23115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:39.387022 23118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:39.386981 23116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:39.387516 23110 server_base.cc:1034] running on GCE node
I20260925 09:18:39.388376 23110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:39.389806 23110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:39.391047 23110 hybrid_clock.cc:648] HybridClock initialized: now 1790327919391012 us; error 47 us; skew 500 ppm
I20260925 09:18:39.391402 23110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:18:39.393826 23110 webserver.cc:460] Webserver started at http://127.3.19.126:36881/ using document root <none> and password file <none>
I20260925 09:18:39.394503 23110 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:39.394656 23110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:39.394970 23110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:39.397876 23110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "519af646a2c043a9b1a8a63df4978724"
format_stamp: "Formatted at 2026-09-25 09:18:39 on dist-test-slave-l6qh"
server_key: "999034242d16110fa622d8c5afc30aa4"
server_key_iv: "5186ad0c37debe9fb28aee838398ba2f"
server_key_version: "encryptionkey@0"
I20260925 09:18:39.398630 23110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "519af646a2c043a9b1a8a63df4978724"
format_stamp: "Formatted at 2026-09-25 09:18:39 on dist-test-slave-l6qh"
server_key: "999034242d16110fa622d8c5afc30aa4"
server_key_iv: "5186ad0c37debe9fb28aee838398ba2f"
server_key_version: "encryptionkey@0"
I20260925 09:18:39.403635 23110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:18:39.406936 23124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:39.408179 23110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:18:39.408356 23110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "519af646a2c043a9b1a8a63df4978724"
format_stamp: "Formatted at 2026-09-25 09:18:39 on dist-test-slave-l6qh"
server_key: "999034242d16110fa622d8c5afc30aa4"
server_key_iv: "5186ad0c37debe9fb28aee838398ba2f"
server_key_version: "encryptionkey@0"
I20260925 09:18:39.408565 23110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:39.436246 23110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:39.437109 23110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:39.437485 23110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:39.455695 23110 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:41271
I20260925 09:18:39.455715 23175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:41271 every 8 connection(s)
I20260925 09:18:39.457015 23110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:39.460744 23176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:39.464221 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 23110
I20260925 09:18:39.464411 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:39.465003 3149 external_mini_cluster.cc:1428] Setting key b3ba1e0e073c3b258c08f2ef85e9208e
I20260925 09:18:39.473069 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724: Bootstrap starting.
I20260925 09:18:39.476227 23176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:39.477504 23176 log.cc:824] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:39.477749 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:39.467057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.3.19.126:41271)
0925 09:18:39.468022 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:39.468032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:39.468073 (+ 41us) server_negotiation.cc:407] Connection header received
0925 09:18:39.469697 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:39.469716 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:39.470143 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:39.470523 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:39.471073 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:39.472259 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:39.473636 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:39.474250 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:39.474439 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:39.475522 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:39.475548 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:39.475569 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:39.476057 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:39.476161 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:39.476181 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:39.476309 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:39.476510 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 09:18:39.476816 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":335,"threads_started":1}
I20260925 09:18:39.480472 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724: No bootstrap required, opened a new log
I20260925 09:18:39.488236 23176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519af646a2c043a9b1a8a63df4978724" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 41271 } }
I20260925 09:18:39.488600 23176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:39.488719 23176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519af646a2c043a9b1a8a63df4978724, State: Initialized, Role: FOLLOWER
I20260925 09:18:39.489393 23176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519af646a2c043a9b1a8a63df4978724" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 41271 } }
I20260925 09:18:39.489701 23176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:39.489856 23176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:39.490015 23176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:39.492094 23176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519af646a2c043a9b1a8a63df4978724" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 41271 } }
I20260925 09:18:39.492621 23176 leader_election.cc:302] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 519af646a2c043a9b1a8a63df4978724; no voters:
I20260925 09:18:39.493209 23176 leader_election.cc:288] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:39.493398 23181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:39.494297 23181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [term 1 LEADER]: Becoming Leader. State: Replica: 519af646a2c043a9b1a8a63df4978724, State: Running, Role: LEADER
I20260925 09:18:39.494796 23181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519af646a2c043a9b1a8a63df4978724" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 41271 } }
I20260925 09:18:39.495486 23176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:39.497861 23183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519af646a2c043a9b1a8a63df4978724. Latest consensus state: current_term: 1 leader_uuid: "519af646a2c043a9b1a8a63df4978724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519af646a2c043a9b1a8a63df4978724" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 41271 } } }
I20260925 09:18:39.498150 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "519af646a2c043a9b1a8a63df4978724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519af646a2c043a9b1a8a63df4978724" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 41271 } } }
I20260925 09:18:39.498376 23183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:39.498443 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:39.499694 23190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:39.504825 23190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:39.513648 23190 catalog_manager.cc:1347] Generated new cluster ID: 456bed5858584e4b833a2c5e34dcba84
I20260925 09:18:39.513770 23190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:39.524672 23190 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:39.525934 23190 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:39.536130 23190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724: Generated new TSK 0
I20260925 09:18:39.536872 23190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:18:39.543859 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.65:0
--local_ip_for_outbound_sockets=127.3.19.65
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:41271
--builtin_ntp_servers=127.3.19.84:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 09:18:39.911777 23200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:39.912035 23200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:39.912191 23200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:39.919425 23200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:39.919553 23200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:39.919742 23200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.65
I20260925 09:18:39.928685 23200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.19.65
--webserver_port=0
--tserver_master_addrs=127.3.19.126:41271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.65
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:39.930083 23200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:39.931839 23200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:39.944680 23205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:39.944788 23208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:39.944757 23206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:39.945662 23200 server_base.cc:1034] running on GCE node
I20260925 09:18:39.946277 23200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:39.947145 23200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:39.948381 23200 hybrid_clock.cc:648] HybridClock initialized: now 1790327919948353 us; error 51 us; skew 500 ppm
I20260925 09:18:39.948697 23200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:18:39.950959 23200 webserver.cc:460] Webserver started at http://127.3.19.65:33193/ using document root <none> and password file <none>
I20260925 09:18:39.951637 23200 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:39.951763 23200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:39.952078 23200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:39.954946 23200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/data/instance:
uuid: "8afb87f8dac14bd7acb857f78063957f"
format_stamp: "Formatted at 2026-09-25 09:18:39 on dist-test-slave-l6qh"
server_key: "c73d3ae468ee3f9b9fb5b69316721442"
server_key_iv: "0b0c3e84420342d5e4bd84b53ddccf85"
server_key_version: "encryptionkey@0"
I20260925 09:18:39.955741 23200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance:
uuid: "8afb87f8dac14bd7acb857f78063957f"
format_stamp: "Formatted at 2026-09-25 09:18:39 on dist-test-slave-l6qh"
server_key: "c73d3ae468ee3f9b9fb5b69316721442"
server_key_iv: "0b0c3e84420342d5e4bd84b53ddccf85"
server_key_version: "encryptionkey@0"
I20260925 09:18:39.960618 23200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 09:18:39.964015 23214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:39.965325 23200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:18:39.965489 23200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/wal
uuid: "8afb87f8dac14bd7acb857f78063957f"
format_stamp: "Formatted at 2026-09-25 09:18:39 on dist-test-slave-l6qh"
server_key: "c73d3ae468ee3f9b9fb5b69316721442"
server_key_iv: "0b0c3e84420342d5e4bd84b53ddccf85"
server_key_version: "encryptionkey@0"
I20260925 09:18:39.965709 23200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:39.986296 23200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:39.987116 23200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:39.987391 23200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:39.988339 23200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:39.989784 23200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:39.989905 23200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:39.990023 23200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:39.990110 23200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:40.023576 23200 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.65:35689
I20260925 09:18:40.023645 23326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.65:35689 every 8 connection(s)
I20260925 09:18:40.024930 23200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/data/info.pb
I20260925 09:18:40.034214 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 23200
I20260925 09:18:40.034399 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/wal/instance
I20260925 09:18:40.034904 3149 external_mini_cluster.cc:1428] Setting key ed1710ce42c415b1b59f9cb93c583e68
I20260925 09:18:40.038007 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:40.027544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.65:60949 (local address 127.3.19.126:41271)
0925 09:18:40.027985 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:40.027989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:40.029292 (+ 1303us) server_negotiation.cc:407] Connection header received
0925 09:18:40.031092 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:40.031096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:40.031169 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:40.031265 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:40.032277 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.032970 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.034683 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.035330 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.035463 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:40.037080 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:40.037087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:40.037091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:40.037111 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:40.037176 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:40.037192 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:40.037310 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:40.037520 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 09:18:40.037612 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":222,"threads_started":1}
I20260925 09:18:40.038201 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.66:0
--local_ip_for_outbound_sockets=127.3.19.66
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:41271
--builtin_ntp_servers=127.3.19.84:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 09:18:40.038961 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:40.028149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:41271 (local address 127.3.19.65:60949)
0925 09:18:40.029107 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:40.029146 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:40.030849 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:40.031462 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:40.031470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:40.031554 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:40.032134 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:40.032151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.033095 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.033099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:40.034543 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:40.034555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.035600 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.035608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:40.035906 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:40.036697 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:18:40.036702 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:40.036767 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:18:40.036879 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:40.037308 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:40.037321 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:40.037331 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 09:18:40.037481 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 09:18:40.037770 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":310,"threads_started":1}
I20260925 09:18:40.040460 23327 heartbeater.cc:347] Connected to a master server at 127.3.19.126:41271
I20260925 09:18:40.040834 23327 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:40.041769 23327 heartbeater.cc:511] Master 127.3.19.126:41271 requested a full tablet report, sending...
I20260925 09:18:40.044090 23141 ts_manager.cc:194] Registered new tserver with Master: 8afb87f8dac14bd7acb857f78063957f (127.3.19.65:35689)
I20260925 09:18:40.045746 23141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.19.65:60949
W20260925 09:18:40.367446 23331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:40.367726 23331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:40.367810 23331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:40.375283 23331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:40.375396 23331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:40.375563 23331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.66
I20260925 09:18:40.384673 23331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.19.66
--webserver_port=0
--tserver_master_addrs=127.3.19.126:41271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.66
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:40.385991 23331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:40.387753 23331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:40.400885 23337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:40.400964 23339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:40.400928 23336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:40.401753 23331 server_base.cc:1034] running on GCE node
I20260925 09:18:40.402390 23331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:40.403332 23331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:40.404561 23331 hybrid_clock.cc:648] HybridClock initialized: now 1790327920404519 us; error 62 us; skew 500 ppm
I20260925 09:18:40.404882 23331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:18:40.407042 23331 webserver.cc:460] Webserver started at http://127.3.19.66:38189/ using document root <none> and password file <none>
I20260925 09:18:40.407732 23331 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:40.407861 23331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:40.408181 23331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:40.411267 23331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/data/instance:
uuid: "6dc205e979384ca5a8a8fee0801ee42e"
format_stamp: "Formatted at 2026-09-25 09:18:40 on dist-test-slave-l6qh"
server_key: "929a8e054d96d93b17c04d1b6548de36"
server_key_iv: "dd4f5ae9af6867f012967a2b2df1bd4a"
server_key_version: "encryptionkey@0"
I20260925 09:18:40.412204 23331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance:
uuid: "6dc205e979384ca5a8a8fee0801ee42e"
format_stamp: "Formatted at 2026-09-25 09:18:40 on dist-test-slave-l6qh"
server_key: "929a8e054d96d93b17c04d1b6548de36"
server_key_iv: "dd4f5ae9af6867f012967a2b2df1bd4a"
server_key_version: "encryptionkey@0"
I20260925 09:18:40.417400 23331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 09:18:40.420760 23345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:40.422062 23331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:18:40.422245 23331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/wal
uuid: "6dc205e979384ca5a8a8fee0801ee42e"
format_stamp: "Formatted at 2026-09-25 09:18:40 on dist-test-slave-l6qh"
server_key: "929a8e054d96d93b17c04d1b6548de36"
server_key_iv: "dd4f5ae9af6867f012967a2b2df1bd4a"
server_key_version: "encryptionkey@0"
I20260925 09:18:40.422477 23331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:40.450493 23331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:40.451393 23331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:40.451674 23331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:40.452657 23331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:40.454156 23331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:40.454276 23331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:40.454412 23331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:40.454500 23331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:40.488626 23331 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.66:40585
I20260925 09:18:40.488711 23457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.66:40585 every 8 connection(s)
I20260925 09:18:40.490049 23331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/data/info.pb
I20260925 09:18:40.496168 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 23331
I20260925 09:18:40.496348 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/wal/instance
I20260925 09:18:40.496933 3149 external_mini_cluster.cc:1428] Setting key b8b0a42f67bcf3113dea67314f62f41c
I20260925 09:18:40.499701 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.19.67:0
--local_ip_for_outbound_sockets=127.3.19.67
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:41271
--builtin_ntp_servers=127.3.19.84:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 09:18:40.503834 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:40.492859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.66:57339 (local address 127.3.19.126:41271)
0925 09:18:40.493870 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:40.493875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:40.494544 (+ 669us) server_negotiation.cc:407] Connection header received
0925 09:18:40.496452 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:40.496458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:40.496545 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:40.496661 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:40.497883 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.498726 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.500429 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.501173 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.501349 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:40.502854 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:40.502863 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:40.502870 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:40.502895 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:40.502937 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:40.502955 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:40.503052 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:40.503350 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 09:18:40.503463 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856}
I20260925 09:18:40.504809 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:40.493368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:41271 (local address 127.3.19.66:57339)
0925 09:18:40.494380 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:40.494419 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:40.496165 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:40.496818 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:40.496826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:40.496908 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:40.497726 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:40.497749 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.498881 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.498889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:40.500287 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:40.500299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.501471 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.501476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:40.501687 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:40.502518 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:18:40.502520 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:40.502566 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:18:40.502653 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:40.503126 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:40.503135 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:40.503142 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 09:18:40.503311 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 09:18:40.503626 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":406,"threads_started":1}
I20260925 09:18:40.506340 23458 heartbeater.cc:347] Connected to a master server at 127.3.19.126:41271
I20260925 09:18:40.506669 23458 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:40.507560 23458 heartbeater.cc:511] Master 127.3.19.126:41271 requested a full tablet report, sending...
I20260925 09:18:40.509120 23141 ts_manager.cc:194] Registered new tserver with Master: 6dc205e979384ca5a8a8fee0801ee42e (127.3.19.66:40585)
I20260925 09:18:40.509984 23141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.19.66:57339
W20260925 09:18:40.837967 23461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:40.838305 23461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:40.838397 23461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:40.845826 23461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:40.845950 23461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:40.846060 23461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.19.67
I20260925 09:18:40.855324 23461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.19.67
--webserver_port=0
--tserver_master_addrs=127.3.19.126:41271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.19.67
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:40.856637 23461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:40.858584 23461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:40.872575 23466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:40.872529 23467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:40.872637 23469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:40.874639 23461 server_base.cc:1034] running on GCE node
I20260925 09:18:40.875303 23461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:40.876376 23461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:40.877640 23461 hybrid_clock.cc:648] HybridClock initialized: now 1790327920877564 us; error 71 us; skew 500 ppm
I20260925 09:18:40.878067 23461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 09:18:40.880442 23461 webserver.cc:460] Webserver started at http://127.3.19.67:38285/ using document root <none> and password file <none>
I20260925 09:18:40.881188 23461 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:40.881350 23461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:40.881683 23461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:40.884598 23461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/data/instance:
uuid: "39a5edb683754296a83761576adc1780"
format_stamp: "Formatted at 2026-09-25 09:18:40 on dist-test-slave-l6qh"
server_key: "b22cc0e055713fb4527410b543f3410f"
server_key_iv: "2125650ba27dfd2d4cc2e23504b7537a"
server_key_version: "encryptionkey@0"
I20260925 09:18:40.885568 23461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance:
uuid: "39a5edb683754296a83761576adc1780"
format_stamp: "Formatted at 2026-09-25 09:18:40 on dist-test-slave-l6qh"
server_key: "b22cc0e055713fb4527410b543f3410f"
server_key_iv: "2125650ba27dfd2d4cc2e23504b7537a"
server_key_version: "encryptionkey@0"
I20260925 09:18:40.891033 23461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:18:40.894901 23475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:40.896508 23461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 09:18:40.896718 23461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/wal
uuid: "39a5edb683754296a83761576adc1780"
format_stamp: "Formatted at 2026-09-25 09:18:40 on dist-test-slave-l6qh"
server_key: "b22cc0e055713fb4527410b543f3410f"
server_key_iv: "2125650ba27dfd2d4cc2e23504b7537a"
server_key_version: "encryptionkey@0"
I20260925 09:18:40.896972 23461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:40.914045 23461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:40.914861 23461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:40.915140 23461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:40.916101 23461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 09:18:40.917646 23461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 09:18:40.917776 23461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:40.917898 23461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 09:18:40.917984 23461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:40.952667 23461 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.67:34845
I20260925 09:18:40.952755 23587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.67:34845 every 8 connection(s)
I20260925 09:18:40.954108 23461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/data/info.pb
I20260925 09:18:40.959702 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 23461
I20260925 09:18:40.959936 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/wal/instance
I20260925 09:18:40.960422 3149 external_mini_cluster.cc:1428] Setting key 9806eaca7f5b159e785e3a9f69d96b25
I20260925 09:18:40.969051 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:40.957014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:36467 (local address 127.3.19.126:41271)
0925 09:18:40.957318 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:40.957322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:40.958863 (+ 1541us) server_negotiation.cc:407] Connection header received
0925 09:18:40.961075 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:40.961082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:40.961205 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:40.961391 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:40.962763 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.963640 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.965540 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.966020 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.966183 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:40.968004 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:40.968016 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:40.968019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:40.968045 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:40.968080 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:40.968099 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:40.968207 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:40.968493 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 09:18:40.968615 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 09:18:40.970288 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:40.957625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:41271 (local address 127.3.19.67:36467)
0925 09:18:40.958662 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:40.958709 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:40.960743 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:40.961691 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:40.961701 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:40.961814 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:18:40.962573 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:40.962594 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.963825 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.963829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:40.965337 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:40.965352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.966290 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.966297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:40.966572 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:40.967473 (+ 901us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 09:18:40.967476 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:18:40.967524 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 09:18:40.967662 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:18:40.968209 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:18:40.968224 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:18:40.968234 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 09:18:40.968456 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 09:18:40.968975 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":362,"threads_started":1}
I20260925 09:18:40.972103 23588 heartbeater.cc:347] Connected to a master server at 127.3.19.126:41271
I20260925 09:18:40.972464 23588 heartbeater.cc:464] Registering TS with master...
I20260925 09:18:40.973402 23588 heartbeater.cc:511] Master 127.3.19.126:41271 requested a full tablet report, sending...
I20260925 09:18:40.975117 23141 ts_manager.cc:194] Registered new tserver with Master: 39a5edb683754296a83761576adc1780 (127.3.19.67:34845)
I20260925 09:18:40.976220 23141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.19.67:36467
I20260925 09:18:40.976383 3149 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 09:18:40.986773 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:40.980697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.3.19.126:41271)
0925 09:18:40.980858 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:40.980864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:40.981390 (+ 526us) server_negotiation.cc:407] Connection header received
0925 09:18:40.981598 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:40.981603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:40.981682 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:40.981793 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:40.982294 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.983013 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.984382 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:40.984802 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:40.984941 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:40.986074 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:40.986079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:40.986082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:40.986100 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:40.986135 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:40.986148 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:40.986246 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:40.986375 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 09:18:40.986469 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 09:18:40.990865 23141 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 09:18:40.998569 23141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 09:18:41.034814 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.013332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:40585 (local address 127.0.0.1:52196)
0925 09:18:41.014453 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:41.014491 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:41.014676 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:41.016279 (+ 1603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:41.016291 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:41.016347 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:41.016922 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:41.016943 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.020972 (+ 4029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.020983 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:41.028604 (+ 7621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:41.028636 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.033736 (+ 5100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.033742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:41.033979 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.034047 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 09:18:41.034151 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 09:18:41.034405 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":704,"threads_started":1}
I20260925 09:18:41.036305 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.013802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.3.19.66:40585)
0925 09:18:41.015392 (+ 1590us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:41.015406 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:41.015447 (+ 41us) server_negotiation.cc:407] Connection header received
0925 09:18:41.015590 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:41.015602 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:41.015938 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:41.017269 (+ 1331us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:41.017595 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.019991 (+ 2396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.030713 (+ 10722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.033517 (+ 2804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.035438 (+ 1921us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.035728 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:41.035826 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 09:18:41.036004 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1353,"thread_start_us":1365,"threads_started":1}
I20260925 09:18:41.041118 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.014260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:35689 (local address 127.0.0.1:34280)
0925 09:18:41.029424 (+ 15164us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:41.029449 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:41.029610 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:41.031051 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:41.031058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:41.031101 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:41.031807 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:41.031827 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.034740 (+ 2913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.034744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:41.038088 (+ 3344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:41.038111 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.040283 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.040289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:41.040475 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.040510 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 09:18:41.040603 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 09:18:41.040755 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15001,"thread_start_us":14320,"threads_started":1}
I20260925 09:18:41.041290 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.014929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.3.19.65:35689)
0925 09:18:41.020792 (+ 5863us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:41.020806 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:41.029801 (+ 8995us) server_negotiation.cc:407] Connection header received
0925 09:18:41.029958 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:41.029970 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:41.030381 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:41.030662 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:41.032101 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.034009 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.038387 (+ 4378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.040045 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.040244 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.040609 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:41.040684 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 09:18:41.040859 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5608,"thread_start_us":5426,"threads_started":1}
I20260925 09:18:41.040763 23393 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3cf48657cc46d694cb02a8a517b944 (DEFAULT_TABLE table=test-table [id=aec844a3d2f245839d63b44e815379be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:41.042747 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.013755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.67:34845 (local address 127.0.0.1:60992)
0925 09:18:41.028167 (+ 14412us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:41.028201 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:41.028432 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:41.034205 (+ 5773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:41.034209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:41.034237 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:41.034570 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:41.034583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.037414 (+ 2831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.037420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:41.039717 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:41.039736 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.041919 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.041926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:41.042095 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.042128 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 09:18:41.042210 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 09:18:41.042354 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14156,"thread_start_us":15311,"threads_started":1}
I20260925 09:18:41.043092 23393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3cf48657cc46d694cb02a8a517b944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:41.043802 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.014008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.3.19.67:34845)
0925 09:18:41.033585 (+ 19577us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:41.033595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:41.033641 (+ 46us) server_negotiation.cc:407] Connection header received
0925 09:18:41.033745 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:41.033754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:41.034014 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:41.034228 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:41.035682 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.037216 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.039990 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.041491 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.041838 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.043124 (+ 1286us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:41.043220 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 09:18:41.043404 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19384,"thread_start_us":19213,"threads_started":1}
I20260925 09:18:41.046103 23262 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3cf48657cc46d694cb02a8a517b944 (DEFAULT_TABLE table=test-table [id=aec844a3d2f245839d63b44e815379be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:41.047806 23262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3cf48657cc46d694cb02a8a517b944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:41.048108 23523 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3cf48657cc46d694cb02a8a517b944 (DEFAULT_TABLE table=test-table [id=aec844a3d2f245839d63b44e815379be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 09:18:41.049000 23327 heartbeater.cc:503] Master 127.3.19.126:41271 was elected leader, sending a full tablet report...
I20260925 09:18:41.049964 23523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3cf48657cc46d694cb02a8a517b944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:41.064316 23606 tablet_bootstrap.cc:492] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e: Bootstrap starting.
I20260925 09:18:41.066743 23607 tablet_bootstrap.cc:492] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f: Bootstrap starting.
I20260925 09:18:41.069416 23608 tablet_bootstrap.cc:492] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780: Bootstrap starting.
I20260925 09:18:41.070073 23606 tablet_bootstrap.cc:654] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:41.071223 23607 tablet_bootstrap.cc:654] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:41.071663 23606 log.cc:824] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:41.072474 23607 log.cc:824] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:41.073115 23608 tablet_bootstrap.cc:654] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:41.074383 23608 log.cc:824] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:41.075740 23606 tablet_bootstrap.cc:492] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e: No bootstrap required, opened a new log
I20260925 09:18:41.075875 23607 tablet_bootstrap.cc:492] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f: No bootstrap required, opened a new log
I20260925 09:18:41.076046 23606 ts_tablet_manager.cc:1397] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260925 09:18:41.076107 23607 ts_tablet_manager.cc:1397] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 09:18:41.076946 23608 tablet_bootstrap.cc:492] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780: No bootstrap required, opened a new log
I20260925 09:18:41.077205 23608 ts_tablet_manager.cc:1397] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 09:18:41.085292 23608 raft_consensus.cc:348] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc205e979384ca5a8a8fee0801ee42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 40585 } } peers { permanent_uuid: "39a5edb683754296a83761576adc1780" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34845 } } peers { permanent_uuid: "8afb87f8dac14bd7acb857f78063957f" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 35689 } }
I20260925 09:18:41.085747 23608 raft_consensus.cc:374] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:41.085894 23608 raft_consensus.cc:729] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a5edb683754296a83761576adc1780, State: Initialized, Role: FOLLOWER
I20260925 09:18:41.086583 23608 consensus_queue.cc:260] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc205e979384ca5a8a8fee0801ee42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 40585 } } peers { permanent_uuid: "39a5edb683754296a83761576adc1780" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34845 } } peers { permanent_uuid: "8afb87f8dac14bd7acb857f78063957f" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 35689 } }
I20260925 09:18:41.087229 23606 raft_consensus.cc:348] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc205e979384ca5a8a8fee0801ee42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 40585 } } peers { permanent_uuid: "39a5edb683754296a83761576adc1780" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34845 } } peers { permanent_uuid: "8afb87f8dac14bd7acb857f78063957f" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 35689 } }
I20260925 09:18:41.087236 23607 raft_consensus.cc:348] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc205e979384ca5a8a8fee0801ee42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 40585 } } peers { permanent_uuid: "39a5edb683754296a83761576adc1780" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34845 } } peers { permanent_uuid: "8afb87f8dac14bd7acb857f78063957f" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 35689 } }
I20260925 09:18:41.087716 23607 raft_consensus.cc:374] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:41.087786 23588 heartbeater.cc:503] Master 127.3.19.126:41271 was elected leader, sending a full tablet report...
I20260925 09:18:41.087785 23606 raft_consensus.cc:374] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:41.087873 23607 raft_consensus.cc:729] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afb87f8dac14bd7acb857f78063957f, State: Initialized, Role: FOLLOWER
I20260925 09:18:41.087919 23606 raft_consensus.cc:729] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc205e979384ca5a8a8fee0801ee42e, State: Initialized, Role: FOLLOWER
I20260925 09:18:41.088547 23608 ts_tablet_manager.cc:1428] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260925 09:18:41.088661 23607 consensus_queue.cc:260] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc205e979384ca5a8a8fee0801ee42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 40585 } } peers { permanent_uuid: "39a5edb683754296a83761576adc1780" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34845 } } peers { permanent_uuid: "8afb87f8dac14bd7acb857f78063957f" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 35689 } }
I20260925 09:18:41.088661 23606 consensus_queue.cc:260] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc205e979384ca5a8a8fee0801ee42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 40585 } } peers { permanent_uuid: "39a5edb683754296a83761576adc1780" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34845 } } peers { permanent_uuid: "8afb87f8dac14bd7acb857f78063957f" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 35689 } }
I20260925 09:18:41.090101 23458 heartbeater.cc:503] Master 127.3.19.126:41271 was elected leader, sending a full tablet report...
I20260925 09:18:41.090847 23606 ts_tablet_manager.cc:1428] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260925 09:18:41.098493 23607 ts_tablet_manager.cc:1428] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f: Time spent starting tablet: real 0.022s user 0.015s sys 0.006s
W20260925 09:18:41.205931 23589 tablet.cc:2369] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:18:41.242453 23459 tablet.cc:2369] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 09:18:41.277539 23328 tablet.cc:2369] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:18:41.418308 23612 raft_consensus.cc:482] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 09:18:41.418697 23612 raft_consensus.cc:504] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc205e979384ca5a8a8fee0801ee42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 40585 } } peers { permanent_uuid: "39a5edb683754296a83761576adc1780" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34845 } } peers { permanent_uuid: "8afb87f8dac14bd7acb857f78063957f" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 35689 } }
I20260925 09:18:41.420527 23612 leader_election.cc:288] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dc205e979384ca5a8a8fee0801ee42e (127.3.19.66:40585), 8afb87f8dac14bd7acb857f78063957f (127.3.19.65:35689)
I20260925 09:18:41.427834 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.420889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.65:35689 (local address 127.3.19.67:52541)
0925 09:18:41.421197 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:41.421223 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:41.421415 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:41.421922 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:41.421925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:41.421948 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:41.422205 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:41.422216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.423770 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.423775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:41.425470 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:41.425482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.427001 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.427007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:41.427186 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.427224 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 09:18:41.427344 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 09:18:41.427488 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":20}
I20260925 09:18:41.427836 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.421140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:52541 (local address 127.3.19.65:35689)
0925 09:18:41.421373 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:41.421380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:41.421406 (+ 26us) server_negotiation.cc:407] Connection header received
0925 09:18:41.421612 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:41.421639 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:41.421749 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:41.421873 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:41.422465 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.423621 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.425680 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.426844 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.426954 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.427047 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:41.427394 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 09:18:41.427500 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 09:18:41.428618 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.421188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.19.67:46557 (local address 127.3.19.66:40585)
0925 09:18:41.421440 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:41.421448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:41.422011 (+ 563us) server_negotiation.cc:407] Connection header received
0925 09:18:41.422205 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:41.422211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:41.422333 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:41.422420 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 09:18:41.423282 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.424200 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.426385 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.427456 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.427576 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.427669 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:41.428152 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 09:18:41.428319 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 09:18:41.429208 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.420956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.66:40585 (local address 127.3.19.67:46557)
0925 09:18:41.421883 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0925 09:18:41.421907 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 09:18:41.422052 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:18:41.422501 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:18:41.422506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:18:41.422601 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 09:18:41.423064 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:41.423080 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.424394 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.424398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:41.426196 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:18:41.426214 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.427651 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.427659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:18:41.427897 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.427934 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 09:18:41.428822 (+ 888us) client_negotiation.cc:240] Negotiation successful
0925 09:18:41.428950 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":3,"thread_start_us":587,"threads_started":1}
I20260925 09:18:41.429188 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3cf48657cc46d694cb02a8a517b944" candidate_uuid: "39a5edb683754296a83761576adc1780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afb87f8dac14bd7acb857f78063957f" is_pre_election: true
I20260925 09:18:41.429881 23282 raft_consensus.cc:2454] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a5edb683754296a83761576adc1780 in term 0.
I20260925 09:18:41.430471 23413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3cf48657cc46d694cb02a8a517b944" candidate_uuid: "39a5edb683754296a83761576adc1780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc205e979384ca5a8a8fee0801ee42e" is_pre_election: true
I20260925 09:18:41.430681 23478 leader_election.cc:302] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a5edb683754296a83761576adc1780, 8afb87f8dac14bd7acb857f78063957f; no voters:
I20260925 09:18:41.431103 23612 raft_consensus.cc:2781] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 09:18:41.431049 23413 raft_consensus.cc:2454] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a5edb683754296a83761576adc1780 in term 0.
I20260925 09:18:41.431272 23612 raft_consensus.cc:482] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 09:18:41.431396 23612 raft_consensus.cc:3037] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:41.433725 23612 raft_consensus.cc:504] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc205e979384ca5a8a8fee0801ee42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 40585 } } peers { permanent_uuid: "39a5edb683754296a83761576adc1780" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34845 } } peers { permanent_uuid: "8afb87f8dac14bd7acb857f78063957f" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 35689 } }
I20260925 09:18:41.434348 23612 leader_election.cc:288] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [CANDIDATE]: Term 1 election: Requested vote from peers 6dc205e979384ca5a8a8fee0801ee42e (127.3.19.66:40585), 8afb87f8dac14bd7acb857f78063957f (127.3.19.65:35689)
I20260925 09:18:41.434878 23413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3cf48657cc46d694cb02a8a517b944" candidate_uuid: "39a5edb683754296a83761576adc1780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc205e979384ca5a8a8fee0801ee42e"
I20260925 09:18:41.435060 23413 raft_consensus.cc:3037] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:41.437527 23413 raft_consensus.cc:2454] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a5edb683754296a83761576adc1780 in term 1.
I20260925 09:18:41.437536 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3cf48657cc46d694cb02a8a517b944" candidate_uuid: "39a5edb683754296a83761576adc1780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afb87f8dac14bd7acb857f78063957f"
I20260925 09:18:41.437763 23282 raft_consensus.cc:3037] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:41.438023 23477 leader_election.cc:302] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a5edb683754296a83761576adc1780, 6dc205e979384ca5a8a8fee0801ee42e; no voters:
I20260925 09:18:41.438370 23612 raft_consensus.cc:2781] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:41.438902 23612 raft_consensus.cc:686] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [term 1 LEADER]: Becoming Leader. State: Replica: 39a5edb683754296a83761576adc1780, State: Running, Role: LEADER
I20260925 09:18:41.439541 23612 consensus_queue.cc:237] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc205e979384ca5a8a8fee0801ee42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 40585 } } peers { permanent_uuid: "39a5edb683754296a83761576adc1780" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34845 } } peers { permanent_uuid: "8afb87f8dac14bd7acb857f78063957f" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 35689 } }
I20260925 09:18:41.441104 23282 raft_consensus.cc:2454] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a5edb683754296a83761576adc1780 in term 1.
I20260925 09:18:41.445163 23141 catalog_manager.cc:5495] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39a5edb683754296a83761576adc1780 (127.3.19.67). New cstate: current_term: 1 leader_uuid: "39a5edb683754296a83761576adc1780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc205e979384ca5a8a8fee0801ee42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 40585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39a5edb683754296a83761576adc1780" member_type: VOTER last_known_addr { host: "127.3.19.67" port: 34845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8afb87f8dac14bd7acb857f78063957f" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 35689 } health_report { overall_health: UNKNOWN } } }
I20260925 09:18:41.503497 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.496075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.3.19.67:34845)
0925 09:18:41.496337 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:41.496346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:41.496916 (+ 570us) server_negotiation.cc:407] Connection header received
0925 09:18:41.497153 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:41.497159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:41.497302 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:41.497459 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:18:41.498067 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.498808 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.500848 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.501431 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.501528 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.502173 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:18:41.502945 (+ 772us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:18:41.503076 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:41.503228 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 09:18:41.503303 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 09:18:41.512099 23413 raft_consensus.cc:1261] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e [term 1 FOLLOWER]: Refusing update from remote peer 39a5edb683754296a83761576adc1780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:18:41.512099 23282 raft_consensus.cc:1261] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f [term 1 FOLLOWER]: Refusing update from remote peer 39a5edb683754296a83761576adc1780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:18:41.513298 23616 consensus_queue.cc:1035] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8afb87f8dac14bd7acb857f78063957f" member_type: VOTER last_known_addr { host: "127.3.19.65" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:18:41.513761 23612 consensus_queue.cc:1035] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc205e979384ca5a8a8fee0801ee42e" member_type: VOTER last_known_addr { host: "127.3.19.66" port: 40585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:18:41.528985 23620 mvcc.cc:205] Tried to move back new op lower bound from 7333183166503100416 to 7333183166221418496. Current Snapshot: MvccSnapshot[applied={T|T < 7333183166503100416}]
I20260925 09:18:41.531802 23621 mvcc.cc:205] Tried to move back new op lower bound from 7333183166503100416 to 7333183166221418496. Current Snapshot: MvccSnapshot[applied={T|T < 7333183166503100416}]
I20260925 09:18:41.537119 23622 mvcc.cc:205] Tried to move back new op lower bound from 7333183166503100416 to 7333183166221418496. Current Snapshot: MvccSnapshot[applied={T|T < 7333183166503100416}]
I20260925 09:18:41.548393 23141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:18:41.548674 23141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 09:18:41.553794 23141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 519af646a2c043a9b1a8a63df4978724: Sending DeleteTablet for 3 replicas of tablet 2b3cf48657cc46d694cb02a8a517b944
I20260925 09:18:41.555389 23393 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3cf48657cc46d694cb02a8a517b944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:41 UTC) from {username='slave'} at 127.0.0.1:52196
I20260925 09:18:41.555689 23523 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3cf48657cc46d694cb02a8a517b944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:41 UTC) from {username='slave'} at 127.0.0.1:60992
I20260925 09:18:41.556293 23262 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3cf48657cc46d694cb02a8a517b944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 09:18:41 UTC) from {username='slave'} at 127.0.0.1:34280
I20260925 09:18:41.557122 23630 tablet_replica.cc:330] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780: stopping tablet replica
I20260925 09:18:41.557488 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 23200
I20260925 09:18:41.557785 23630 raft_consensus.cc:2229] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [term 1 LEADER]: Raft consensus shutting down.
I20260925 09:18:41.558624 23630 raft_consensus.cc:2258] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:41.562250 23630 ts_tablet_manager.cc:1905] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:41.575436 23631 tablet_replica.cc:330] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f: stopping tablet replica
I20260925 09:18:41.575932 23630 ts_tablet_manager.cc:1918] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:41.576268 23630 log.cc:1196] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-2/wal/wals/2b3cf48657cc46d694cb02a8a517b944
I20260925 09:18:41.577054 23630 ts_tablet_manager.cc:1939] T 2b3cf48657cc46d694cb02a8a517b944 P 39a5edb683754296a83761576adc1780: Deleting consensus metadata
I20260925 09:18:41.577389 23631 raft_consensus.cc:2229] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:18:41.577965 23631 raft_consensus.cc:2258] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:41.578475 23629 tablet_replica.cc:330] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e: stopping tablet replica
I20260925 09:18:41.579264 23629 raft_consensus.cc:2229] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 09:18:41.579900 23629 raft_consensus.cc:2258] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 09:18:41.580732 23125 catalog_manager.cc:4841] TS 39a5edb683754296a83761576adc1780 (127.3.19.67:34845): tablet 2b3cf48657cc46d694cb02a8a517b944 (table test-table [id=aec844a3d2f245839d63b44e815379be]) successfully deleted
I20260925 09:18:41.582490 23631 ts_tablet_manager.cc:1905] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:41.585040 23629 ts_tablet_manager.cc:1905] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 09:18:41.590451 23631 ts_tablet_manager.cc:1918] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:41.590703 23631 log.cc:1196] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-0/wal/wals/2b3cf48657cc46d694cb02a8a517b944
I20260925 09:18:41.591171 23629 ts_tablet_manager.cc:1918] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 09:18:41.591380 23629 log.cc:1196] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e: Deleting WAL directory at /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790327600440376-3149-0/minicluster-data/ts-1/wal/wals/2b3cf48657cc46d694cb02a8a517b944
I20260925 09:18:41.591410 23631 ts_tablet_manager.cc:1939] T 2b3cf48657cc46d694cb02a8a517b944 P 8afb87f8dac14bd7acb857f78063957f: Deleting consensus metadata
I20260925 09:18:41.591956 23629 ts_tablet_manager.cc:1939] T 2b3cf48657cc46d694cb02a8a517b944 P 6dc205e979384ca5a8a8fee0801ee42e: Deleting consensus metadata
I20260925 09:18:41.593303 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.560045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.3.19.65:35689)
0925 09:18:41.560274 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:41.560283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:41.570089 (+ 9806us) server_negotiation.cc:407] Connection header received
0925 09:18:41.570611 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:41.570618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:41.570815 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:41.570909 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:41.571595 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.572845 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.578187 (+ 5342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.579123 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.579252 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.589897 (+ 10645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:41.589915 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:41.589937 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:41.591823 (+ 1886us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:41.591911 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:41.591945 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:41.592092 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:41.592781 (+ 689us) server_negotiation.cc:299] Negotiation successful
0925 09:18:41.592897 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 09:18:41.593976 23127 catalog_manager.cc:4841] TS 8afb87f8dac14bd7acb857f78063957f (127.3.19.65:35689): tablet 2b3cf48657cc46d694cb02a8a517b944 (table test-table [id=aec844a3d2f245839d63b44e815379be]) successfully deleted
I20260925 09:18:41.594097 23126 catalog_manager.cc:4841] TS 6dc205e979384ca5a8a8fee0801ee42e (127.3.19.66:40585): tablet 2b3cf48657cc46d694cb02a8a517b944 (table test-table [id=aec844a3d2f245839d63b44e815379be]) successfully deleted
I20260925 09:18:41.594202 23322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:41.703372 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 23200
I20260925 09:18:41.728119 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 23331
I20260925 09:18:41.735980 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.729052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.3.19.66:40585)
0925 09:18:41.729287 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:41.729294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:41.729318 (+ 24us) server_negotiation.cc:407] Connection header received
0925 09:18:41.729483 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:41.729489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:41.729605 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:41.729722 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:41.730068 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.731199 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.732443 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.733112 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.733363 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.734350 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:41.734362 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:41.734382 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:41.735188 (+ 806us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:41.735252 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:41.735284 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:41.735391 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:41.735481 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 09:18:41.735588 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 09:18:41.736723 23453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:41.826682 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 23331
I20260925 09:18:41.844384 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 23461
I20260925 09:18:41.851640 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:41.845284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.3.19.67:34845)
0925 09:18:41.845471 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:41.845477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:41.845501 (+ 24us) server_negotiation.cc:407] Connection header received
0925 09:18:41.845693 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:41.845704 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:41.845801 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:41.845923 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:41.846362 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.847180 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.848424 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:41.848868 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:41.849028 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:41.850209 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:41.850217 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:41.850231 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:41.850860 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:41.850908 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:41.850927 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:41.851065 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:41.851192 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 09:18:41.851304 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 09:18:41.852415 23583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:41.972033 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 23461
I20260925 09:18:41.992151 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 23110
I20260925 09:18:41.993505 23171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:42.103066 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 23110
2026-09-25T09:18:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3153 ms)
[----------] 6 tests from EncryptionPolicyTest (21906 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:18:42 dist-test-slave-l6qh krb5kdc[23639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:18:42 dist-test-slave-l6qh krb5kdc[23639](info): set up 2 sockets
Sep 25 09:18:42 dist-test-slave-l6qh krb5kdc[23639](info): commencing operation
krb5kdc: starting...
W20260925 09:18:43.274794 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.004s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:18:43 dist-test-slave-l6qh krb5kdc[23639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:18:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 09:18:43.564566 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:43.916399 23652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:43.916702 23652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:43.916869 23652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:43.924459 23652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:43.924576 23652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:43.924722 23652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:43.924852 23652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:43.924930 23652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:43.935067 23652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:43.936450 23652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:43.937820 23652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:43.950196 23657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:43.950125 23658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:43.950232 23660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:43.950937 23652 server_base.cc:1034] running on GCE node
I20260925 09:18:43.951834 23652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:43.953374 23652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:43.954648 23652 hybrid_clock.cc:648] HybridClock initialized: now 1790327923954578 us; error 47 us; skew 500 ppm
I20260925 09:18:43.955035 23652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:43 dist-test-slave-l6qh krb5kdc[23639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:43.958317 23652 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 09:18:43.959925 23652 webserver.cc:460] Webserver started at http://127.0.0.1:33085/ using document root <none> and password file <none>
I20260925 09:18:43.960650 23652 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:43.960795 23652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:43.961102 23652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:43.964192 23652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "40e84a086f85423f9b5a935ce38446dc"
format_stamp: "Formatted at 2026-09-25 09:18:43 on dist-test-slave-l6qh"
server_key: "7ad3779b887bf0e570e5598673e24ac2"
server_key_iv: "fc292432db86d9e6414a1462d6a0abe2"
server_key_version: "encryptionkey@0"
I20260925 09:18:43.964954 23652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "40e84a086f85423f9b5a935ce38446dc"
format_stamp: "Formatted at 2026-09-25 09:18:43 on dist-test-slave-l6qh"
server_key: "7ad3779b887bf0e570e5598673e24ac2"
server_key_iv: "fc292432db86d9e6414a1462d6a0abe2"
server_key_version: "encryptionkey@0"
I20260925 09:18:43.970247 23652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 09:18:43.973877 23667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:43.975385 23652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:18:43.975603 23652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "40e84a086f85423f9b5a935ce38446dc"
format_stamp: "Formatted at 2026-09-25 09:18:43 on dist-test-slave-l6qh"
server_key: "7ad3779b887bf0e570e5598673e24ac2"
server_key_iv: "fc292432db86d9e6414a1462d6a0abe2"
server_key_version: "encryptionkey@0"
I20260925 09:18:43.975862 23652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:44.003700 23652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:44.008023 23652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:44.008325 23652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:44.028756 23652 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40871
I20260925 09:18:44.028822 23718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40871 every 8 connection(s)
I20260925 09:18:44.030405 23652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:44.033604 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 23652
I20260925 09:18:44.033830 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:44.034384 23719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:44.034483 3149 external_mini_cluster.cc:1428] Setting key 50f95db1a251dacf5acf73ac59c860e8
I20260925 09:18:44.049743 23719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc: Bootstrap starting.
Sep 25 09:18:44 dist-test-slave-l6qh krb5kdc[23639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 09:18:44.053102 23719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:44.054225 23719 log.cc:824] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:44.057550 23719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc: No bootstrap required, opened a new log
I20260925 09:18:44.059422 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:44.036418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34137 (local address 127.0.0.1:40871)
0925 09:18:44.037368 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:44.037379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:44.037427 (+ 48us) server_negotiation.cc:407] Connection header received
0925 09:18:44.039175 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:44.039197 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:44.039652 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:44.040004 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:44.041985 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:44.043313 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:44.045200 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:44.046034 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:44.046199 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:44.051425 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:44.051434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:44.051450 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:44.051505 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:44.054733 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:44.055618 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:44.055625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:44.055631 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:44.055786 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:44.056339 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:44.056343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:44.056346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:44.056948 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:44.057219 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:44.057909 (+ 690us) server_negotiation.cc:299] Negotiation successful
0925 09:18:44.058314 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":426,"threads_started":1}
I20260925 09:18:44.063994 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:18:44.066731 23719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e84a086f85423f9b5a935ce38446dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40871 } }
I20260925 09:18:44.067227 23719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:44.067335 23719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e84a086f85423f9b5a935ce38446dc, State: Initialized, Role: FOLLOWER
I20260925 09:18:44.068082 23719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e84a086f85423f9b5a935ce38446dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40871 } }
I20260925 09:18:44.068430 23719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:44.068670 23719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:44.068857 23719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:44.072263 23719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e84a086f85423f9b5a935ce38446dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40871 } }
I20260925 09:18:44.072953 23719 leader_election.cc:302] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40e84a086f85423f9b5a935ce38446dc; no voters:
I20260925 09:18:44.073774 23719 leader_election.cc:288] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:44.073931 23729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:44.075098 23729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [term 1 LEADER]: Becoming Leader. State: Replica: 40e84a086f85423f9b5a935ce38446dc, State: Running, Role: LEADER
I20260925 09:18:44.075848 23729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e84a086f85423f9b5a935ce38446dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40871 } }
I20260925 09:18:44.076069 23719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:44.081315 23731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40e84a086f85423f9b5a935ce38446dc. Latest consensus state: current_term: 1 leader_uuid: "40e84a086f85423f9b5a935ce38446dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e84a086f85423f9b5a935ce38446dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40871 } } }
I20260925 09:18:44.081758 23731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:44.082501 23730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40e84a086f85423f9b5a935ce38446dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e84a086f85423f9b5a935ce38446dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40871 } } }
I20260925 09:18:44.082820 23730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:44.086966 23738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:44.089891 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:44.070322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49221 (local address 127.0.0.1:40871)
0925 09:18:44.070543 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:44.070584 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:44.070872 (+ 288us) server_negotiation.cc:407] Connection header received
0925 09:18:44.071099 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:44.071104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:44.071181 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:44.071280 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:44.073313 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:44.074952 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:44.076772 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:44.077619 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:44.077792 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:44.081608 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:44.081620 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:44.081625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:44.081669 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:44.086261 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:44.087326 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:44.087329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:44.087332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:44.087443 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:44.087958 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:44.087962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:44.087964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:44.088326 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:44.088635 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:44.089021 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 09:18:44.089207 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 09:18:44.093811 23738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:44.103237 23738 catalog_manager.cc:1347] Generated new cluster ID: 44a41b3f4cdd446eadadb5d492d6979e
I20260925 09:18:44.103420 23738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:44.126876 23738 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:44.128868 23738 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:44.144034 23738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40e84a086f85423f9b5a935ce38446dc: Generated new TSK 0
I20260925 09:18:44.144927 23738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:18:44.154564 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 23652
I20260925 09:18:44.156047 23714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:44.251204 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 23652
W20260925 09:18:44.270287 23637 connection.cc:535] client connection to 127.0.0.1:40871 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40871): Connection reset by peer (error 104)
2026-09-25T09:18:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2139 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:18:44 dist-test-slave-l6qh krb5kdc[23751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:18:44 dist-test-slave-l6qh krb5kdc[23751](info): set up 2 sockets
Sep 25 09:18:44 dist-test-slave-l6qh krb5kdc[23751](info): commencing operation
krb5kdc: starting...
W20260925 09:18:45.382499 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:18:45 dist-test-slave-l6qh krb5kdc[23751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:18:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 09:18:45.645880 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:45.979959 23764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:45.980324 23764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:45.980479 23764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:45.987746 23764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:45.987900 23764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:45.988001 23764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:45.988107 23764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:45.988183 23764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:45.997939 23764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42683
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:45.999544 23764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:46.000897 23764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:46.012918 23772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:46.013073 23769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:46.012950 23770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:46.013631 23764 server_base.cc:1034] running on GCE node
I20260925 09:18:46.014364 23764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:46.015719 23764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:46.016955 23764 hybrid_clock.cc:648] HybridClock initialized: now 1790327926016915 us; error 48 us; skew 500 ppm
I20260925 09:18:46.017376 23764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:46 dist-test-slave-l6qh krb5kdc[23751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:46.020449 23764 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 09:18:46.022053 23764 webserver.cc:460] Webserver started at http://127.0.0.1:37879/ using document root <none> and password file <none>
I20260925 09:18:46.022724 23764 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:46.022869 23764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:46.023168 23764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:46.026223 23764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "64228b269da1449ea5d690f5c653d7d2"
format_stamp: "Formatted at 2026-09-25 09:18:46 on dist-test-slave-l6qh"
server_key: "03d38726c8e89a4abfbf2e47c8c0c480"
server_key_iv: "9866efc2968c50915b71ecd07406df02"
server_key_version: "encryptionkey@0"
I20260925 09:18:46.027004 23764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "64228b269da1449ea5d690f5c653d7d2"
format_stamp: "Formatted at 2026-09-25 09:18:46 on dist-test-slave-l6qh"
server_key: "03d38726c8e89a4abfbf2e47c8c0c480"
server_key_iv: "9866efc2968c50915b71ecd07406df02"
server_key_version: "encryptionkey@0"
I20260925 09:18:46.032418 23764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:18:46.035851 23779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:46.037096 23764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:18:46.037331 23764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "64228b269da1449ea5d690f5c653d7d2"
format_stamp: "Formatted at 2026-09-25 09:18:46 on dist-test-slave-l6qh"
server_key: "03d38726c8e89a4abfbf2e47c8c0c480"
server_key_iv: "9866efc2968c50915b71ecd07406df02"
server_key_version: "encryptionkey@0"
I20260925 09:18:46.037523 23764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:46.056416 23764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:46.060673 23764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:46.060968 23764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:46.079941 23764 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42683
I20260925 09:18:46.079983 23830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42683 every 8 connection(s)
I20260925 09:18:46.081358 23764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:46.084223 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 23764
I20260925 09:18:46.084440 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:46.084925 3149 external_mini_cluster.cc:1428] Setting key 29f9ad0ce2c2b0609595046de2eaeeaa
I20260925 09:18:46.085114 23831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:46.098495 23834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:46.086811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35997 (local address 127.0.0.1:42683)
0925 09:18:46.087613 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:46.087624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:46.087666 (+ 42us) server_negotiation.cc:407] Connection header received
0925 09:18:46.089630 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:46.089651 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:46.090036 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:46.090373 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:46.090996 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:46.092280 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:46.094166 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:46.094801 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:46.094930 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:46.096286 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:46.096297 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:46.096310 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:46.096847 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:46.096921 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:46.096944 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:46.097057 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:46.097429 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 09:18:46.097613 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":369,"threads_started":1}
I20260925 09:18:46.098784 23831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2: Bootstrap starting.
I20260925 09:18:46.101984 23831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:46.102211 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:18:46.103056 23831 log.cc:824] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:46.105713 23831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2: No bootstrap required, opened a new log
I20260925 09:18:46.113077 23834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:46.106482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46949 (local address 127.0.0.1:42683)
0925 09:18:46.106676 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:46.106684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:46.107142 (+ 458us) server_negotiation.cc:407] Connection header received
0925 09:18:46.107321 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:46.107325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:46.107382 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:46.107463 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:46.108057 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:46.108830 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:46.110624 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:46.111066 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:46.111169 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:46.112311 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:46.112315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:46.112318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:46.112333 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:46.112379 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:46.112395 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:46.112473 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:46.112676 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 09:18:46.112719 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 09:18:46.114980 23831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64228b269da1449ea5d690f5c653d7d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260925 09:18:46.115338 23831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:46.115450 23831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64228b269da1449ea5d690f5c653d7d2, State: Initialized, Role: FOLLOWER
I20260925 09:18:46.116037 23831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64228b269da1449ea5d690f5c653d7d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260925 09:18:46.116294 23831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:46.116443 23831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:46.116624 23831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:46.118911 23831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64228b269da1449ea5d690f5c653d7d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260925 09:18:46.119407 23831 leader_election.cc:302] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64228b269da1449ea5d690f5c653d7d2; no voters:
I20260925 09:18:46.119998 23831 leader_election.cc:288] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:46.120157 23841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:46.120822 23841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [term 1 LEADER]: Becoming Leader. State: Replica: 64228b269da1449ea5d690f5c653d7d2, State: Running, Role: LEADER
I20260925 09:18:46.121382 23841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64228b269da1449ea5d690f5c653d7d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260925 09:18:46.122021 23831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:46.124047 23843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64228b269da1449ea5d690f5c653d7d2. Latest consensus state: current_term: 1 leader_uuid: "64228b269da1449ea5d690f5c653d7d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64228b269da1449ea5d690f5c653d7d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260925 09:18:46.124418 23843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:46.125869 23842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64228b269da1449ea5d690f5c653d7d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64228b269da1449ea5d690f5c653d7d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260925 09:18:46.126303 23842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:46.128522 23847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:46.133981 23847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:46.143291 23847 catalog_manager.cc:1347] Generated new cluster ID: 255a71615d20455fa9fb6f50fbc0f4da
I20260925 09:18:46.143404 23847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:46.154557 23847 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:46.156293 23847 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:46.165825 23847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64228b269da1449ea5d690f5c653d7d2: Generated new TSK 0
I20260925 09:18:46.166675 23847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:18:46.176663 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 23764
W20260925 09:18:46.177753 23826 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35997
W20260925 09:18:46.178516 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 23764: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:18:46.178694 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 23764
W20260925 09:18:46.196457 23749 connection.cc:535] client connection to 127.0.0.1:42683 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42683): Connection reset by peer (error 104)
2026-09-25T09:18:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:18:46 dist-test-slave-l6qh krb5kdc[23862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:18:46 dist-test-slave-l6qh krb5kdc[23862](info): set up 2 sockets
Sep 25 09:18:46 dist-test-slave-l6qh krb5kdc[23862](info): commencing operation
krb5kdc: starting...
W20260925 09:18:47.286468 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:18:47 dist-test-slave-l6qh krb5kdc[23862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:18:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 09:18:47.538882 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33117
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:47.882645 23875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:47.882992 23875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:47.883159 23875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:47.890443 23875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:47.890708 23875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:47.890843 23875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:47.890971 23875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:47.891103 23875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:47.902164 23875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33117
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:47.903720 23875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:47.905105 23875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:47.917367 23880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:47.917418 23881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:47.917471 23883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:47.922686 23875 server_base.cc:1034] running on GCE node
I20260925 09:18:47.923537 23875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:47.924958 23875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:47.926229 23875 hybrid_clock.cc:648] HybridClock initialized: now 1790327927926156 us; error 79 us; skew 500 ppm
I20260925 09:18:47.926627 23875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:47 dist-test-slave-l6qh krb5kdc[23862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:47.929970 23875 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 09:18:47.931699 23875 webserver.cc:460] Webserver started at http://127.0.0.1:40589/ using document root <none> and password file <none>
I20260925 09:18:47.932498 23875 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:47.932664 23875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:47.932989 23875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:47.936249 23875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "448d3110bd754ec08020b956e0539430"
format_stamp: "Formatted at 2026-09-25 09:18:47 on dist-test-slave-l6qh"
server_key: "39b02aa7b6253d7bbbaa5eaa8245930a"
server_key_iv: "05151c6d0b0ac5aad8cac0522bfddacb"
server_key_version: "encryptionkey@0"
I20260925 09:18:47.937106 23875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "448d3110bd754ec08020b956e0539430"
format_stamp: "Formatted at 2026-09-25 09:18:47 on dist-test-slave-l6qh"
server_key: "39b02aa7b6253d7bbbaa5eaa8245930a"
server_key_iv: "05151c6d0b0ac5aad8cac0522bfddacb"
server_key_version: "encryptionkey@0"
I20260925 09:18:47.942512 23875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 09:18:47.946137 23890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:47.947413 23875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:18:47.947630 23875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "448d3110bd754ec08020b956e0539430"
format_stamp: "Formatted at 2026-09-25 09:18:47 on dist-test-slave-l6qh"
server_key: "39b02aa7b6253d7bbbaa5eaa8245930a"
server_key_iv: "05151c6d0b0ac5aad8cac0522bfddacb"
server_key_version: "encryptionkey@0"
I20260925 09:18:47.947865 23875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:47.954172 23875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:47.958312 23875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:47.958632 23875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:47.978874 23875 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33117
I20260925 09:18:47.978924 23941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33117 every 8 connection(s)
I20260925 09:18:47.980324 23875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:47.983939 23942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:47.984309 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 23875
I20260925 09:18:47.984500 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:47.984977 3149 external_mini_cluster.cc:1428] Setting key 139a008d9c0f175191807480a86fb920
I20260925 09:18:47.993552 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:47.986817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51969 (local address 127.0.0.1:33117)
0925 09:18:47.987704 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:47.987715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:47.987762 (+ 47us) server_negotiation.cc:407] Connection header received
0925 09:18:47.990024 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:47.990052 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:47.990408 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:47.990765 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:47.991871 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:47.991883 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:47.991896 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:47.991910 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:47.991975 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:47.991996 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:47.992189 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:47.992316 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 09:18:47.992529 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":421,"threads_started":1}
I20260925 09:18:47.997488 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:18:47.997539 23942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430: Bootstrap starting.
I20260925 09:18:48.000550 23942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:48.001613 23942 log.cc:824] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:48.004830 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:48.002192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51965 (local address 127.0.0.1:33117)
0925 09:18:48.002442 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:48.002452 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:48.002862 (+ 410us) server_negotiation.cc:407] Connection header received
0925 09:18:48.003063 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:48.003069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:48.003133 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:48.003258 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:48.004159 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:48.004166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:48.004173 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:48.004191 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:48.004244 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:48.004272 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:48.004402 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:48.004455 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 09:18:48.004529 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 09:18:48.005105 23942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430: No bootstrap required, opened a new log
I20260925 09:18:48.013554 23942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448d3110bd754ec08020b956e0539430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260925 09:18:48.013959 23942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:48.014092 23942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448d3110bd754ec08020b956e0539430, State: Initialized, Role: FOLLOWER
I20260925 09:18:48.014734 23942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448d3110bd754ec08020b956e0539430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260925 09:18:48.014991 23942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:48.015162 23942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:48.015343 23942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:48.017985 23942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448d3110bd754ec08020b956e0539430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260925 09:18:48.018718 23942 leader_election.cc:302] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 448d3110bd754ec08020b956e0539430; no voters:
I20260925 09:18:48.019333 23942 leader_election.cc:288] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:48.019503 23952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:48.020313 23952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [term 1 LEADER]: Becoming Leader. State: Replica: 448d3110bd754ec08020b956e0539430, State: Running, Role: LEADER
I20260925 09:18:48.020852 23952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448d3110bd754ec08020b956e0539430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260925 09:18:48.021601 23942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:48.023970 23954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 448d3110bd754ec08020b956e0539430. Latest consensus state: current_term: 1 leader_uuid: "448d3110bd754ec08020b956e0539430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448d3110bd754ec08020b956e0539430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } } }
I20260925 09:18:48.024472 23954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:48.025097 23953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "448d3110bd754ec08020b956e0539430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448d3110bd754ec08020b956e0539430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } } }
I20260925 09:18:48.025509 23953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:48.027697 23961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:48.032580 23961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:48.042495 23961 catalog_manager.cc:1347] Generated new cluster ID: 45720fc5f47d41aaabad36c0151fe84d
I20260925 09:18:48.042624 23961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:48.075145 23961 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:48.075580 23961 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:48.082355 23961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 448d3110bd754ec08020b956e0539430: Generated new TSK 0
I20260925 09:18:48.083241 23961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:18:48.088183 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 23875
W20260925 09:18:48.089187 23937 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51969
W20260925 09:18:48.090219 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 23875: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:18:48.090473 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 23875
2026-09-25T09:18:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:18:48 dist-test-slave-l6qh krb5kdc[23973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:18:48 dist-test-slave-l6qh krb5kdc[23973](info): set up 2 sockets
Sep 25 09:18:48 dist-test-slave-l6qh krb5kdc[23973](info): commencing operation
krb5kdc: starting...
W20260925 09:18:49.198523 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:18:49 dist-test-slave-l6qh krb5kdc[23973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:18:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 09:18:49.459993 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:49.799207 23986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:49.799588 23986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:49.799777 23986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:49.807296 23986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:49.807469 23986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:49.807600 23986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:49.807715 23986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:49.807786 23986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:49.818210 23986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:49.819557 23986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:49.820909 23986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:49.833212 23992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:49.833335 23991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:49.833303 23994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:49.834321 23986 server_base.cc:1034] running on GCE node
I20260925 09:18:49.834992 23986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:49.836293 23986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:49.837493 23986 hybrid_clock.cc:648] HybridClock initialized: now 1790327929837461 us; error 28 us; skew 500 ppm
I20260925 09:18:49.837806 23986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:49 dist-test-slave-l6qh krb5kdc[23973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:49.840840 23986 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 09:18:49.842389 23986 webserver.cc:460] Webserver started at http://127.0.0.1:45339/ using document root <none> and password file <none>
I20260925 09:18:49.843108 23986 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:49.843255 23986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:49.843616 23986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:49.846848 23986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "c8b68d3c946d4ea28a7865dc19741d7c"
format_stamp: "Formatted at 2026-09-25 09:18:49 on dist-test-slave-l6qh"
server_key: "6298dbde8c06e26d168db84b1c8c62cc"
server_key_iv: "4b17d7b2f6b1410d73fcca55b5b60cf9"
server_key_version: "encryptionkey@0"
I20260925 09:18:49.847675 23986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "c8b68d3c946d4ea28a7865dc19741d7c"
format_stamp: "Formatted at 2026-09-25 09:18:49 on dist-test-slave-l6qh"
server_key: "6298dbde8c06e26d168db84b1c8c62cc"
server_key_iv: "4b17d7b2f6b1410d73fcca55b5b60cf9"
server_key_version: "encryptionkey@0"
I20260925 09:18:49.853101 23986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 09:18:49.856711 24001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:49.858093 23986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:18:49.858301 23986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "c8b68d3c946d4ea28a7865dc19741d7c"
format_stamp: "Formatted at 2026-09-25 09:18:49 on dist-test-slave-l6qh"
server_key: "6298dbde8c06e26d168db84b1c8c62cc"
server_key_iv: "4b17d7b2f6b1410d73fcca55b5b60cf9"
server_key_version: "encryptionkey@0"
I20260925 09:18:49.858510 23986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:49.894119 23986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:49.898747 23986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:49.899034 23986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:49.917701 23986 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43345
I20260925 09:18:49.917748 24052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43345 every 8 connection(s)
I20260925 09:18:49.919169 23986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:49.919924 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 23986
I20260925 09:18:49.920120 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:49.920652 3149 external_mini_cluster.cc:1428] Setting key 48b2f1f4a62cc8473ca7926136a648e6
I20260925 09:18:49.923553 24053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 09:18:49 dist-test-slave-l6qh krb5kdc[23973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 09:18:49.936941 24053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c: Bootstrap starting.
I20260925 09:18:49.941114 24053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:49.942480 24053 log.cc:824] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:49.944880 24056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:49.922408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46265 (local address 127.0.0.1:43345)
0925 09:18:49.923246 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:49.923258 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:49.923299 (+ 41us) server_negotiation.cc:407] Connection header received
0925 09:18:49.925006 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:49.925027 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:49.925479 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:49.925804 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:49.927618 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:49.929000 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:49.930867 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:49.931524 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:49.931688 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:49.936616 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:49.936627 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:49.936645 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:49.936690 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:49.940673 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:49.941456 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:49.941467 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:49.941476 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:49.941711 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:49.942070 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:49.942075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:49.942078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:49.942778 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:49.943132 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:49.943527 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 09:18:49.943884 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":395,"threads_started":1}
I20260925 09:18:49.945647 24053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c: No bootstrap required, opened a new log
I20260925 09:18:49.948493 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:18:49.953359 24053 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b68d3c946d4ea28a7865dc19741d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43345 } }
I20260925 09:18:49.953836 24053 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:49.953958 24053 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b68d3c946d4ea28a7865dc19741d7c, State: Initialized, Role: FOLLOWER
I20260925 09:18:49.954555 24053 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b68d3c946d4ea28a7865dc19741d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43345 } }
I20260925 09:18:49.954797 24053 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:49.954962 24053 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:49.955148 24053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:49.957486 24053 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b68d3c946d4ea28a7865dc19741d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43345 } }
I20260925 09:18:49.958002 24053 leader_election.cc:302] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8b68d3c946d4ea28a7865dc19741d7c; no voters:
I20260925 09:18:49.958560 24053 leader_election.cc:288] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:49.958706 24063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:49.959343 24063 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [term 1 LEADER]: Becoming Leader. State: Replica: c8b68d3c946d4ea28a7865dc19741d7c, State: Running, Role: LEADER
I20260925 09:18:49.959949 24063 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b68d3c946d4ea28a7865dc19741d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43345 } }
I20260925 09:18:49.961881 24053 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:49.962889 24064 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8b68d3c946d4ea28a7865dc19741d7c. Latest consensus state: current_term: 1 leader_uuid: "c8b68d3c946d4ea28a7865dc19741d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b68d3c946d4ea28a7865dc19741d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43345 } } }
I20260925 09:18:49.963331 24064 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:49.963879 24065 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8b68d3c946d4ea28a7865dc19741d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b68d3c946d4ea28a7865dc19741d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43345 } } }
I20260925 09:18:49.964172 24065 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:49.971040 24056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:49.952878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58683 (local address 127.0.0.1:43345)
0925 09:18:49.953105 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:49.953120 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:49.953524 (+ 404us) server_negotiation.cc:407] Connection header received
0925 09:18:49.953680 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:49.953683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:49.953737 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:49.953842 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:49.955440 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:49.956199 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:49.957556 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:49.958044 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:49.958188 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:49.961581 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:49.961588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:49.961592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:49.961635 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:49.966277 (+ 4642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:49.967496 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:49.967505 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:49.967511 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:49.967797 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:49.968369 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:49.968377 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:49.968383 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:49.968901 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:49.969299 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:49.970054 (+ 755us) server_negotiation.cc:299] Negotiation successful
0925 09:18:49.970373 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 09:18:49.974447 24068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:49.979825 24068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:49.988415 24068 catalog_manager.cc:1347] Generated new cluster ID: 4f6b8f8d3d7a41418244eb1013b9b864
I20260925 09:18:49.988567 24068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:50.012036 24068 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:50.013751 24068 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:50.021149 24068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8b68d3c946d4ea28a7865dc19741d7c: Generated new TSK 0
I20260925 09:18:50.021968 24068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:18:50.029192 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 23986
I20260925 09:18:50.030290 24048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:50.122251 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 23986
2026-09-25T09:18:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2029 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:18:50 dist-test-slave-l6qh krb5kdc[24085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:18:50 dist-test-slave-l6qh krb5kdc[24085](info): set up 2 sockets
Sep 25 09:18:50 dist-test-slave-l6qh krb5kdc[24085](info): commencing operation
krb5kdc: starting...
W20260925 09:18:51.234422 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:18:51 dist-test-slave-l6qh krb5kdc[24085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:18:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 09:18:51.504158 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38869
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:51.861196 24098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:51.861514 24098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:51.861620 24098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:51.868806 24098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:51.868928 24098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:51.868975 24098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:51.869050 24098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:51.869095 24098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:51.879138 24098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:51.880407 24098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:51.881651 24098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:51.893311 24104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:51.893325 24103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:51.893254 24106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:51.894426 24098 server_base.cc:1034] running on GCE node
I20260925 09:18:51.895270 24098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:51.896759 24098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:51.898010 24098 hybrid_clock.cc:648] HybridClock initialized: now 1790327931897934 us; error 51 us; skew 500 ppm
I20260925 09:18:51.898425 24098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:51 dist-test-slave-l6qh krb5kdc[24085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:51.905910 24098 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 09:18:51.907436 24098 webserver.cc:460] Webserver started at http://127.0.0.1:33525/ using document root <none> and password file <none>
I20260925 09:18:51.908159 24098 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:51.908277 24098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:51.908546 24098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:51.911960 24098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "12a3d0fdb4094b6fbc72aceeea06f771"
format_stamp: "Formatted at 2026-09-25 09:18:51 on dist-test-slave-l6qh"
server_key: "5a315aec97eb949b69f0eeb111b0a469"
server_key_iv: "126923eaae7d0fbdb5ee2793ae04ac3f"
server_key_version: "encryptionkey@0"
I20260925 09:18:51.912724 24098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "12a3d0fdb4094b6fbc72aceeea06f771"
format_stamp: "Formatted at 2026-09-25 09:18:51 on dist-test-slave-l6qh"
server_key: "5a315aec97eb949b69f0eeb111b0a469"
server_key_iv: "126923eaae7d0fbdb5ee2793ae04ac3f"
server_key_version: "encryptionkey@0"
I20260925 09:18:51.918223 24098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 09:18:51.922029 24113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:51.923477 24098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:18:51.923667 24098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "12a3d0fdb4094b6fbc72aceeea06f771"
format_stamp: "Formatted at 2026-09-25 09:18:51 on dist-test-slave-l6qh"
server_key: "5a315aec97eb949b69f0eeb111b0a469"
server_key_iv: "126923eaae7d0fbdb5ee2793ae04ac3f"
server_key_version: "encryptionkey@0"
I20260925 09:18:51.923848 24098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:51.946729 24098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:51.951623 24098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:51.951884 24098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:51.972271 24098 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38869
I20260925 09:18:51.972294 24164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38869 every 8 connection(s)
I20260925 09:18:51.973827 24098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:51.977643 24165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:51.981917 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 24098
I20260925 09:18:51.982106 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:51.982648 3149 external_mini_cluster.cc:1428] Setting key 701b70c6bdc1beb143dac49b3b9a8e43
I20260925 09:18:51.988926 24165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771: Bootstrap starting.
I20260925 09:18:51.992141 24165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:51.993304 24165 log.cc:824] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:51.996142 24165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771: No bootstrap required, opened a new log
I20260925 09:18:51.997608 24168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:51.984501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55599 (local address 127.0.0.1:38869)
0925 09:18:51.985438 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:51.985450 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:51.985492 (+ 42us) server_negotiation.cc:407] Connection header received
0925 09:18:51.987930 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:51.987954 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:51.988376 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:51.988810 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:51.989422 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:51.990938 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:51.992598 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:51.993467 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:51.993716 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:51.995043 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:51.995051 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:51.995070 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:51.995664 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:51.995754 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:51.995781 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:51.995926 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:51.996158 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 09:18:51.996470 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":394,"threads_started":1}
I20260925 09:18:52.001760 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:18:52.004794 24165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a3d0fdb4094b6fbc72aceeea06f771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38869 } }
I20260925 09:18:52.005232 24165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:52.005383 24165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a3d0fdb4094b6fbc72aceeea06f771, State: Initialized, Role: FOLLOWER
I20260925 09:18:52.006009 24165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a3d0fdb4094b6fbc72aceeea06f771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38869 } }
I20260925 09:18:52.006263 24165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:52.006412 24165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:52.006626 24165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:52.009639 24165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a3d0fdb4094b6fbc72aceeea06f771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38869 } }
I20260925 09:18:52.010197 24165 leader_election.cc:302] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12a3d0fdb4094b6fbc72aceeea06f771; no voters:
I20260925 09:18:52.010891 24165 leader_election.cc:288] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:52.011018 24175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:52.013325 24165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:52.014571 24168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:52.006405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53443 (local address 127.0.0.1:38869)
0925 09:18:52.006657 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:52.006666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:52.007206 (+ 540us) server_negotiation.cc:407] Connection header received
0925 09:18:52.007392 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:52.007397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:52.007462 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:52.007560 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:52.008170 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:52.008902 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:52.010750 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:52.011808 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:52.012040 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:52.013382 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:52.013393 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:52.013400 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:52.013431 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:52.013493 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:52.013527 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:52.013703 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:52.013888 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 09:18:52.014018 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 09:18:52.017894 24175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [term 1 LEADER]: Becoming Leader. State: Replica: 12a3d0fdb4094b6fbc72aceeea06f771, State: Running, Role: LEADER
I20260925 09:18:52.018534 24175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a3d0fdb4094b6fbc72aceeea06f771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38869 } }
I20260925 09:18:52.021308 24176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12a3d0fdb4094b6fbc72aceeea06f771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a3d0fdb4094b6fbc72aceeea06f771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38869 } } }
I20260925 09:18:52.021813 24176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:52.021700 24177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12a3d0fdb4094b6fbc72aceeea06f771. Latest consensus state: current_term: 1 leader_uuid: "12a3d0fdb4094b6fbc72aceeea06f771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a3d0fdb4094b6fbc72aceeea06f771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38869 } } }
I20260925 09:18:52.022022 24177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:52.026155 24183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:52.031831 24183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:52.041605 24183 catalog_manager.cc:1347] Generated new cluster ID: 964460db80a74fff92ce5f8a6a18ba93
I20260925 09:18:52.041744 24183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:52.068730 24183 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:52.070326 24183 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:52.077209 24183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12a3d0fdb4094b6fbc72aceeea06f771: Generated new TSK 0
I20260925 09:18:52.078060 24183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:18:52.088934 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24098
W20260925 09:18:52.090049 24160 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55599
W20260925 09:18:52.090727 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24098: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:18:52.090862 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 24098
2026-09-25T09:18:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1968 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:18:52 dist-test-slave-l6qh krb5kdc[24196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:18:52 dist-test-slave-l6qh krb5kdc[24196](info): set up 2 sockets
Sep 25 09:18:52 dist-test-slave-l6qh krb5kdc[24196](info): commencing operation
krb5kdc: starting...
W20260925 09:18:53.262580 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:18:53 dist-test-slave-l6qh krb5kdc[24196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:18:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 09:18:53.520143 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:53.855558 24209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:53.855834 24209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:53.855930 24209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:53.863011 24209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:53.863114 24209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:53.863161 24209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:53.863222 24209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:53.863266 24209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:53.872820 24209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:53.874075 24209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:53.875315 24209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:53.887069 24215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:53.887178 24217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:53.887147 24214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:53.887903 24209 server_base.cc:1034] running on GCE node
I20260925 09:18:53.888651 24209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:53.890094 24209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:53.891319 24209 hybrid_clock.cc:648] HybridClock initialized: now 1790327933891257 us; error 52 us; skew 500 ppm
I20260925 09:18:53.891691 24209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:53 dist-test-slave-l6qh krb5kdc[24196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:53.894914 24209 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 09:18:53.896313 24209 webserver.cc:460] Webserver started at http://127.0.0.1:36245/ using document root <none> and password file <none>
I20260925 09:18:53.896979 24209 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:53.897127 24209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:53.897457 24209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:53.900367 24209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "3da8e41e50874c4b9007394d53e753a2"
format_stamp: "Formatted at 2026-09-25 09:18:53 on dist-test-slave-l6qh"
server_key: "41849bd93494bd6135b82c34aad76828"
server_key_iv: "8bbf98fc178239e8ff559570c0957c8d"
server_key_version: "encryptionkey@0"
I20260925 09:18:53.901095 24209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "3da8e41e50874c4b9007394d53e753a2"
format_stamp: "Formatted at 2026-09-25 09:18:53 on dist-test-slave-l6qh"
server_key: "41849bd93494bd6135b82c34aad76828"
server_key_iv: "8bbf98fc178239e8ff559570c0957c8d"
server_key_version: "encryptionkey@0"
I20260925 09:18:53.906508 24209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 09:18:53.909837 24224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:53.911134 24209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:18:53.911314 24209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "3da8e41e50874c4b9007394d53e753a2"
format_stamp: "Formatted at 2026-09-25 09:18:53 on dist-test-slave-l6qh"
server_key: "41849bd93494bd6135b82c34aad76828"
server_key_iv: "8bbf98fc178239e8ff559570c0957c8d"
server_key_version: "encryptionkey@0"
I20260925 09:18:53.911505 24209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:53.917173 24209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:53.921749 24209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:53.922039 24209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:53.940626 24209 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46533
I20260925 09:18:53.940711 24275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46533 every 8 connection(s)
I20260925 09:18:53.942072 24209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:53.944504 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 24209
I20260925 09:18:53.944700 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:53.945312 3149 external_mini_cluster.cc:1428] Setting key 6baeb1f31ebe974b1f92061e80fd4202
I20260925 09:18:53.946403 24276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:53.953567 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:53.947061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48629 (local address 127.0.0.1:46533)
0925 09:18:53.947876 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:53.947890 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:53.947937 (+ 47us) server_negotiation.cc:407] Connection header received
0925 09:18:53.949858 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:53.949885 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:53.950325 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:53.950748 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:53.951687 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:53.951694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:53.951711 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:53.951728 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:53.951823 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:53.951857 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:53.952055 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:53.952194 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 09:18:53.952375 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":337,"threads_started":1}
I20260925 09:18:53.956959 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:18:53.960942 24276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2: Bootstrap starting.
I20260925 09:18:53.964486 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:53.962172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48951 (local address 127.0.0.1:46533)
0925 09:18:53.962364 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:53.962371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:53.962693 (+ 322us) server_negotiation.cc:407] Connection header received
0925 09:18:53.962871 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:53.962874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:53.962916 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:53.963005 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:53.963886 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:53.963890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:53.963893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:53.963905 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:53.963940 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:53.963953 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:53.964036 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:53.964173 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 09:18:53.964215 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 09:18:53.964838 24276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:53.966053 24276 log.cc:824] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:53.968638 24276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2: No bootstrap required, opened a new log
I20260925 09:18:53.975800 24276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da8e41e50874c4b9007394d53e753a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46533 } }
I20260925 09:18:53.976161 24276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:53.976266 24276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da8e41e50874c4b9007394d53e753a2, State: Initialized, Role: FOLLOWER
I20260925 09:18:53.976840 24276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da8e41e50874c4b9007394d53e753a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46533 } }
I20260925 09:18:53.977089 24276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:53.977288 24276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:53.977481 24276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:53.979657 24276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da8e41e50874c4b9007394d53e753a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46533 } }
I20260925 09:18:53.980130 24276 leader_election.cc:302] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3da8e41e50874c4b9007394d53e753a2; no voters:
I20260925 09:18:53.980777 24276 leader_election.cc:288] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:53.980893 24286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:53.981637 24286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [term 1 LEADER]: Becoming Leader. State: Replica: 3da8e41e50874c4b9007394d53e753a2, State: Running, Role: LEADER
I20260925 09:18:53.982158 24286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da8e41e50874c4b9007394d53e753a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46533 } }
I20260925 09:18:53.982770 24276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:53.984807 24287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3da8e41e50874c4b9007394d53e753a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da8e41e50874c4b9007394d53e753a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46533 } } }
I20260925 09:18:53.985270 24287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:53.985139 24288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3da8e41e50874c4b9007394d53e753a2. Latest consensus state: current_term: 1 leader_uuid: "3da8e41e50874c4b9007394d53e753a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da8e41e50874c4b9007394d53e753a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46533 } } }
I20260925 09:18:53.985526 24288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:53.987011 24294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:53.991940 24294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:54.000622 24294 catalog_manager.cc:1347] Generated new cluster ID: 589791e252b648288a34b0f99ab8afad
I20260925 09:18:54.000720 24294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:54.012454 24294 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:54.012887 24294 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:54.021492 24294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3da8e41e50874c4b9007394d53e753a2: Generated new TSK 0
I20260925 09:18:54.022151 24294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:18:54.033479 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24209
W20260925 09:18:54.034503 24271 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48629
W20260925 09:18:54.035211 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24209: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:18:54.035378 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 24209
2026-09-25T09:18:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:18:54 dist-test-slave-l6qh krb5kdc[24307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:18:54 dist-test-slave-l6qh krb5kdc[24307](info): set up 2 sockets
Sep 25 09:18:54 dist-test-slave-l6qh krb5kdc[24307](info): commencing operation
krb5kdc: starting...
W20260925 09:18:55.134690 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:18:55 dist-test-slave-l6qh krb5kdc[24307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:18:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 09:18:55.384649 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:55.715590 24320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:55.715842 24320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:55.715930 24320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:55.722857 24320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:55.722950 24320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:55.722993 24320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:55.723052 24320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:55.723093 24320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:55.732159 24320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:55.733291 24320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:55.734494 24320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:55.746126 24325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:55.747069 24326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:55.747169 24328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:55.748121 24320 server_base.cc:1034] running on GCE node
I20260925 09:18:55.748845 24320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:55.750295 24320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:55.751531 24320 hybrid_clock.cc:648] HybridClock initialized: now 1790327935751467 us; error 45 us; skew 500 ppm
I20260925 09:18:55.751895 24320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:55 dist-test-slave-l6qh krb5kdc[24307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:55.755261 24320 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 09:18:55.756856 24320 webserver.cc:460] Webserver started at http://127.0.0.1:33247/ using document root <none> and password file <none>
I20260925 09:18:55.757630 24320 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:55.757772 24320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:55.758106 24320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:55.761154 24320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "556181561005404e99998c9139edb451"
format_stamp: "Formatted at 2026-09-25 09:18:55 on dist-test-slave-l6qh"
server_key: "944c050a291c7654eeb13334a4566040"
server_key_iv: "4b869452832d0ebaa341756bb710e905"
server_key_version: "encryptionkey@0"
I20260925 09:18:55.762099 24320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "556181561005404e99998c9139edb451"
format_stamp: "Formatted at 2026-09-25 09:18:55 on dist-test-slave-l6qh"
server_key: "944c050a291c7654eeb13334a4566040"
server_key_iv: "4b869452832d0ebaa341756bb710e905"
server_key_version: "encryptionkey@0"
I20260925 09:18:55.767454 24320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:18:55.770949 24335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:55.772192 24320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:18:55.772387 24320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "556181561005404e99998c9139edb451"
format_stamp: "Formatted at 2026-09-25 09:18:55 on dist-test-slave-l6qh"
server_key: "944c050a291c7654eeb13334a4566040"
server_key_iv: "4b869452832d0ebaa341756bb710e905"
server_key_version: "encryptionkey@0"
I20260925 09:18:55.772609 24320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:55.817900 24320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:55.822059 24320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:55.822372 24320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:55.842425 24320 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36895
I20260925 09:18:55.842442 24386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36895 every 8 connection(s)
I20260925 09:18:55.843904 24320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:55.846647 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 24320
I20260925 09:18:55.846895 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:55.847374 3149 external_mini_cluster.cc:1428] Setting key be662f2003365c7ec49b191e8e7c4a6a
I20260925 09:18:55.847808 24387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:55.861869 24387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451: Bootstrap starting.
Sep 25 09:18:55 dist-test-slave-l6qh krb5kdc[24307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 09:18:55.865403 24387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:55.866446 24387 log.cc:824] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:55.869760 24387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451: No bootstrap required, opened a new log
I20260925 09:18:55.871903 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:55.849296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:38895 (local address 127.0.0.1:36895)
0925 09:18:55.850079 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:55.850094 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:55.850139 (+ 45us) server_negotiation.cc:407] Connection header received
0925 09:18:55.852569 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:55.852595 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:55.853116 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:55.853543 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:55.854756 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:55.856846 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:55.858152 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:55.859239 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:55.859443 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:55.863961 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:55.863968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:55.863981 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:55.864014 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:55.867254 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:55.868172 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:55.868179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:55.868186 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:55.868414 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:55.868847 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:55.868852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:55.868855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:55.869503 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:55.869838 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:55.870221 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 09:18:55.870585 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":327,"threads_started":1}
I20260925 09:18:55.875787 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:18:55.878839 24387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556181561005404e99998c9139edb451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36895 } }
I20260925 09:18:55.879236 24387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:55.879405 24387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556181561005404e99998c9139edb451, State: Initialized, Role: FOLLOWER
I20260925 09:18:55.879968 24387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556181561005404e99998c9139edb451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36895 } }
I20260925 09:18:55.880285 24387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:55.880460 24387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:55.880671 24387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:55.882963 24387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556181561005404e99998c9139edb451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36895 } }
I20260925 09:18:55.883438 24387 leader_election.cc:302] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 556181561005404e99998c9139edb451; no voters:
I20260925 09:18:55.884090 24387 leader_election.cc:288] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:55.884186 24397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:55.885766 24397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [term 1 LEADER]: Becoming Leader. State: Replica: 556181561005404e99998c9139edb451, State: Running, Role: LEADER
I20260925 09:18:55.886387 24397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556181561005404e99998c9139edb451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36895 } }
I20260925 09:18:55.886502 24387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:55.890677 24398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "556181561005404e99998c9139edb451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556181561005404e99998c9139edb451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36895 } } }
I20260925 09:18:55.891062 24398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:55.890901 24399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 556181561005404e99998c9139edb451. Latest consensus state: current_term: 1 leader_uuid: "556181561005404e99998c9139edb451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556181561005404e99998c9139edb451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36895 } } }
I20260925 09:18:55.891297 24399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:55.898739 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:55.880535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:47599 (local address 127.0.0.1:36895)
0925 09:18:55.880741 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:55.880749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:55.881140 (+ 391us) server_negotiation.cc:407] Connection header received
0925 09:18:55.881371 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:55.881377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:55.881452 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:55.881559 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:55.882915 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:55.883816 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:55.885537 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:55.886265 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:55.886399 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:55.888972 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:55.888978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:55.888981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:18:55.889009 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:55.893500 (+ 4491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:55.894722 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:55.894729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:55.894735 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:55.894923 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:18:55.895677 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:18:55.895684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:18:55.895690 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:18:55.896240 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:55.897769 (+ 1529us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:55.897991 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 09:18:55.898222 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 09:18:55.900444 24406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:55.904904 24406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:55.913009 24406 catalog_manager.cc:1347] Generated new cluster ID: ff4f1fd786d146d4a60098d6b937e033
I20260925 09:18:55.913106 24406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:55.953848 24406 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:55.955682 24406 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:55.965058 24406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 556181561005404e99998c9139edb451: Generated new TSK 0
I20260925 09:18:55.966058 24406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:18:55.984714 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24320
I20260925 09:18:55.985863 24382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:18:56.078438 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 24320
W20260925 09:18:56.096424 24305 connection.cc:535] client connection to 127.0.0.1:36895 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36895): Connection reset by peer (error 104)
2026-09-25T09:18:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2045 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:18:56 dist-test-slave-l6qh krb5kdc[24419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:18:56 dist-test-slave-l6qh krb5kdc[24419](info): set up 2 sockets
Sep 25 09:18:56 dist-test-slave-l6qh krb5kdc[24419](info): commencing operation
krb5kdc: starting...
W20260925 09:18:57.178624 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:18:57 dist-test-slave-l6qh krb5kdc[24419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:18:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 09:18:57.444772 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:57.777973 24432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:57.778316 24432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:57.778482 24432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:57.785584 24432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:57.785745 24432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:57.785838 24432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:57.785945 24432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:57.786060 24432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:57.796213 24432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:57.797784 24432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:57.799147 24432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:57.811648 24437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:57.811648 24440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:57.811623 24438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:57.812176 24432 server_base.cc:1034] running on GCE node
I20260925 09:18:57.813053 24432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:57.814481 24432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:57.815701 24432 hybrid_clock.cc:648] HybridClock initialized: now 1790327937815657 us; error 40 us; skew 500 ppm
I20260925 09:18:57.816051 24432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:57 dist-test-slave-l6qh krb5kdc[24419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:57.823172 24432 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 09:18:57.824797 24432 webserver.cc:460] Webserver started at http://127.0.0.1:40477/ using document root <none> and password file <none>
I20260925 09:18:57.825585 24432 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:57.825735 24432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:57.826030 24432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:57.829387 24432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "19a1a4574bd6409582e548b652e698df"
format_stamp: "Formatted at 2026-09-25 09:18:57 on dist-test-slave-l6qh"
server_key: "3d8a05ea7a493958b05ae58032bc0b5f"
server_key_iv: "72d29963077ee4addb50c88916b599f2"
server_key_version: "encryptionkey@0"
I20260925 09:18:57.830183 24432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "19a1a4574bd6409582e548b652e698df"
format_stamp: "Formatted at 2026-09-25 09:18:57 on dist-test-slave-l6qh"
server_key: "3d8a05ea7a493958b05ae58032bc0b5f"
server_key_iv: "72d29963077ee4addb50c88916b599f2"
server_key_version: "encryptionkey@0"
I20260925 09:18:57.835248 24432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:18:57.838788 24447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:57.840163 24432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:18:57.840358 24432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "19a1a4574bd6409582e548b652e698df"
format_stamp: "Formatted at 2026-09-25 09:18:57 on dist-test-slave-l6qh"
server_key: "3d8a05ea7a493958b05ae58032bc0b5f"
server_key_iv: "72d29963077ee4addb50c88916b599f2"
server_key_version: "encryptionkey@0"
I20260925 09:18:57.840557 24432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:57.858193 24432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:57.862686 24432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:57.862963 24432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:57.881784 24432 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33369
I20260925 09:18:57.881825 24498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33369 every 8 connection(s)
I20260925 09:18:57.883119 24432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:57.886812 24499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:57.889353 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 24432
I20260925 09:18:57.889566 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:57.890013 3149 external_mini_cluster.cc:1428] Setting key 17a02fc0506313729a70cfaa18962175
I20260925 09:18:57.898540 24499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df: Bootstrap starting.
I20260925 09:18:57.902153 24499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:57.903152 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:57.891744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:44245 (local address 127.0.0.1:33369)
0925 09:18:57.892541 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:57.892552 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:57.892593 (+ 41us) server_negotiation.cc:407] Connection header received
0925 09:18:57.894251 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:57.894273 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:57.894644 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:57.894979 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:57.895627 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:57.896926 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:57.898791 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:57.899450 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:57.899576 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:57.900909 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:57.900916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:57.900929 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:57.901492 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:57.901563 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:57.901591 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:57.901728 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:57.902009 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 09:18:57.902206 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":408,"threads_started":1}
I20260925 09:18:57.903223 24499 log.cc:824] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:57.906351 24499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df: No bootstrap required, opened a new log
I20260925 09:18:57.906917 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:18:57.915275 24499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a1a4574bd6409582e548b652e698df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } }
I20260925 09:18:57.915751 24499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:57.915891 24499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a1a4574bd6409582e548b652e698df, State: Initialized, Role: FOLLOWER
I20260925 09:18:57.916499 24499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a1a4574bd6409582e548b652e698df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } }
I20260925 09:18:57.916743 24499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:57.916891 24499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:57.917080 24499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:57.918026 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:57.911114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:42143 (local address 127.0.0.1:33369)
0925 09:18:57.911321 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:57.911327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:57.911627 (+ 300us) server_negotiation.cc:407] Connection header received
0925 09:18:57.911813 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:57.911817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:57.911874 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:57.911946 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:57.912649 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:57.913486 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:57.915279 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:18:57.915808 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:18:57.915948 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:18:57.917161 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:57.917185 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:57.917190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:57.917213 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:57.917298 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:57.917323 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:57.917409 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:57.917622 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 09:18:57.917663 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 09:18:57.920023 24499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a1a4574bd6409582e548b652e698df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } }
I20260925 09:18:57.920499 24499 leader_election.cc:302] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19a1a4574bd6409582e548b652e698df; no voters:
I20260925 09:18:57.921036 24499 leader_election.cc:288] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:57.921164 24509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:57.921913 24509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [term 1 LEADER]: Becoming Leader. State: Replica: 19a1a4574bd6409582e548b652e698df, State: Running, Role: LEADER
I20260925 09:18:57.922408 24509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a1a4574bd6409582e548b652e698df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } }
I20260925 09:18:57.923094 24499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:57.925625 24511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19a1a4574bd6409582e548b652e698df. Latest consensus state: current_term: 1 leader_uuid: "19a1a4574bd6409582e548b652e698df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a1a4574bd6409582e548b652e698df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } } }
I20260925 09:18:57.925601 24510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19a1a4574bd6409582e548b652e698df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a1a4574bd6409582e548b652e698df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } } }
I20260925 09:18:57.926091 24511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:57.926179 24510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:57.928243 24515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:57.933876 24515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:57.942472 24515 catalog_manager.cc:1347] Generated new cluster ID: be49dc542ff547c99ccc0d9553223bbe
I20260925 09:18:57.942592 24515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:57.956694 24515 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:57.958483 24515 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:57.974845 24515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19a1a4574bd6409582e548b652e698df: Generated new TSK 0
I20260925 09:18:57.975857 24515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:18:57.986824 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24432
W20260925 09:18:57.987919 24494 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.32:44245
W20260925 09:18:57.988657 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24432: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:18:57.988817 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 24432
W20260925 09:18:58.006610 24417 connection.cc:535] client connection to 127.0.0.1:33369 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33369): Connection reset by peer (error 104)
2026-09-25T09:18:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:18:58 dist-test-slave-l6qh krb5kdc[24530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:18:58 dist-test-slave-l6qh krb5kdc[24530](info): set up 2 sockets
Sep 25 09:18:58 dist-test-slave-l6qh krb5kdc[24530](info): commencing operation
krb5kdc: starting...
W20260925 09:18:59.098623 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:18:59 dist-test-slave-l6qh krb5kdc[24530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:18:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:18:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 09:18:59.356478 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:18:59.681870 24543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:18:59.682188 24543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:18:59.682282 24543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:18:59.690752 24543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:18:59.690855 24543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:18:59.690900 24543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:18:59.690961 24543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:18:59.691004 24543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:18:59.700743 24543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:18:59.701982 24543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:18:59.703239 24543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:18:59.715253 24549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:59.715332 24548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:18:59.715283 24551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:18:59.715916 24543 server_base.cc:1034] running on GCE node
I20260925 09:18:59.716735 24543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:18:59.718139 24543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:18:59.719374 24543 hybrid_clock.cc:648] HybridClock initialized: now 1790327939719322 us; error 39 us; skew 500 ppm
I20260925 09:18:59.719731 24543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:18:59 dist-test-slave-l6qh krb5kdc[24530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:18:59.723120 24543 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 09:18:59.724560 24543 webserver.cc:460] Webserver started at http://127.0.0.1:41987/ using document root <none> and password file <none>
I20260925 09:18:59.725274 24543 fs_manager.cc:359] Metadata directory not provided
I20260925 09:18:59.725430 24543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:18:59.725737 24543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:18:59.728610 24543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "28b9700ddb1641ecafa4202311419415"
format_stamp: "Formatted at 2026-09-25 09:18:59 on dist-test-slave-l6qh"
server_key: "52ed1806b218942a8e72631d6810b86f"
server_key_iv: "c657e5454c5c310ac6ac1237d7fecd3a"
server_key_version: "encryptionkey@0"
I20260925 09:18:59.729490 24543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "28b9700ddb1641ecafa4202311419415"
format_stamp: "Formatted at 2026-09-25 09:18:59 on dist-test-slave-l6qh"
server_key: "52ed1806b218942a8e72631d6810b86f"
server_key_iv: "c657e5454c5c310ac6ac1237d7fecd3a"
server_key_version: "encryptionkey@0"
I20260925 09:18:59.734513 24543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 09:18:59.737898 24558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:18:59.739146 24543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:18:59.739336 24543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "28b9700ddb1641ecafa4202311419415"
format_stamp: "Formatted at 2026-09-25 09:18:59 on dist-test-slave-l6qh"
server_key: "52ed1806b218942a8e72631d6810b86f"
server_key_iv: "c657e5454c5c310ac6ac1237d7fecd3a"
server_key_version: "encryptionkey@0"
I20260925 09:18:59.739527 24543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:18:59.745188 24543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:18:59.749471 24543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:18:59.749791 24543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:18:59.768189 24543 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41061
I20260925 09:18:59.768265 24609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41061 every 8 connection(s)
I20260925 09:18:59.769570 24543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:18:59.770311 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 24543
I20260925 09:18:59.770498 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:18:59.771075 3149 external_mini_cluster.cc:1428] Setting key 78c7322c9832be00a4584937423a9245
I20260925 09:18:59.773829 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:18:59.778481 24613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:59.772820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:59963 (local address 127.0.0.1:41061)
0925 09:18:59.773586 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:59.773597 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:59.773635 (+ 38us) server_negotiation.cc:407] Connection header received
0925 09:18:59.775251 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:59.775271 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:59.775574 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:59.775927 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:59.776920 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:59.776928 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:59.776940 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:59.776953 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:59.777024 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:59.777042 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:59.777289 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:59.777438 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 09:18:59.777603 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":357,"threads_started":1}
I20260925 09:18:59.782061 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:18:59.785475 24610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415: Bootstrap starting.
I20260925 09:18:59.788776 24613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:18:59.786404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:49339 (local address 127.0.0.1:41061)
0925 09:18:59.786558 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 09:18:59.786566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:18:59.786964 (+ 398us) server_negotiation.cc:407] Connection header received
0925 09:18:59.787139 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:18:59.787145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:18:59.787201 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:18:59.787315 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:18:59.788161 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:18:59.788166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:18:59.788171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:18:59.788188 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:18:59.788231 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:18:59.788245 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:18:59.788364 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 09:18:59.788478 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 09:18:59.788531 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 09:18:59.788928 24610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415: Neither blocks nor log segments found. Creating new log.
I20260925 09:18:59.790102 24610 log.cc:824] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415: Log is configured to *not* fsync() on all Append() calls
I20260925 09:18:59.792520 24610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415: No bootstrap required, opened a new log
I20260925 09:18:59.800635 24610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b9700ddb1641ecafa4202311419415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41061 } }
I20260925 09:18:59.800974 24610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:18:59.801081 24610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b9700ddb1641ecafa4202311419415, State: Initialized, Role: FOLLOWER
I20260925 09:18:59.801685 24610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b9700ddb1641ecafa4202311419415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41061 } }
I20260925 09:18:59.801899 24610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:18:59.802040 24610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:18:59.802208 24610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:18:59.804286 24610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b9700ddb1641ecafa4202311419415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41061 } }
I20260925 09:18:59.804793 24610 leader_election.cc:302] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28b9700ddb1641ecafa4202311419415; no voters:
I20260925 09:18:59.805375 24610 leader_election.cc:288] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:18:59.805511 24620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:18:59.806720 24620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [term 1 LEADER]: Becoming Leader. State: Replica: 28b9700ddb1641ecafa4202311419415, State: Running, Role: LEADER
I20260925 09:18:59.807199 24620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b9700ddb1641ecafa4202311419415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41061 } }
I20260925 09:18:59.807394 24610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:18:59.810529 24622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28b9700ddb1641ecafa4202311419415. Latest consensus state: current_term: 1 leader_uuid: "28b9700ddb1641ecafa4202311419415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b9700ddb1641ecafa4202311419415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41061 } } }
I20260925 09:18:59.810616 24621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28b9700ddb1641ecafa4202311419415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b9700ddb1641ecafa4202311419415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41061 } } }
I20260925 09:18:59.810943 24622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:59.810962 24621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415 [sys.catalog]: This master's current role is: LEADER
I20260925 09:18:59.812898 24629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:18:59.818032 24629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:18:59.826159 24629 catalog_manager.cc:1347] Generated new cluster ID: 697df05f5fe943efa7cc1c44caae3d88
I20260925 09:18:59.826268 24629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:18:59.846686 24629 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:18:59.847179 24629 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:18:59.856688 24629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28b9700ddb1641ecafa4202311419415: Generated new TSK 0
I20260925 09:18:59.857652 24629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:18:59.868685 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24543
W20260925 09:18:59.869705 24605 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.32:59963
W20260925 09:18:59.870405 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24543: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:18:59.870566 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 24543
2026-09-25T09:18:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:18:59 dist-test-slave-l6qh krb5kdc[24641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:18:59 dist-test-slave-l6qh krb5kdc[24641](info): set up 2 sockets
Sep 25 09:18:59 dist-test-slave-l6qh krb5kdc[24641](info): commencing operation
krb5kdc: starting...
W20260925 09:19:00.986637 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:19:01 dist-test-slave-l6qh krb5kdc[24641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:19:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:19:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 09:19:01.249612 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41649
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:19:01.582386 24654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:01.582718 24654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:01.582877 24654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:01.590029 24654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:01.590171 24654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:01.590279 24654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:01.590379 24654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:01.590458 24654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:01.599845 24654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41649
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:01.601084 24654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:01.602409 24654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:19:01.614459 24659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:01.614499 24662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:01.614539 24660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:01.615438 24654 server_base.cc:1034] running on GCE node
I20260925 09:19:01.616266 24654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:01.617729 24654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:01.618961 24654 hybrid_clock.cc:648] HybridClock initialized: now 1790327941618925 us; error 62 us; skew 500 ppm
I20260925 09:19:01.619335 24654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:01 dist-test-slave-l6qh krb5kdc[24641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:01.626338 24654 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 09:19:01.628211 24654 webserver.cc:460] Webserver started at http://127.0.0.1:44677/ using document root <none> and password file <none>
I20260925 09:19:01.628976 24654 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:01.629129 24654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:01.629498 24654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:01.632578 24654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "8db81ff777e04798a309b26ac5fe2342"
format_stamp: "Formatted at 2026-09-25 09:19:01 on dist-test-slave-l6qh"
server_key: "608894cebbc0bd4be730fcf0730b4df1"
server_key_iv: "fc9f71f11297cd98c9d66c96e75d91ff"
server_key_version: "encryptionkey@0"
I20260925 09:19:01.633453 24654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "8db81ff777e04798a309b26ac5fe2342"
format_stamp: "Formatted at 2026-09-25 09:19:01 on dist-test-slave-l6qh"
server_key: "608894cebbc0bd4be730fcf0730b4df1"
server_key_iv: "fc9f71f11297cd98c9d66c96e75d91ff"
server_key_version: "encryptionkey@0"
I20260925 09:19:01.638967 24654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 09:19:01.642674 24669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:01.644096 24654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:19:01.644304 24654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "8db81ff777e04798a309b26ac5fe2342"
format_stamp: "Formatted at 2026-09-25 09:19:01 on dist-test-slave-l6qh"
server_key: "608894cebbc0bd4be730fcf0730b4df1"
server_key_iv: "fc9f71f11297cd98c9d66c96e75d91ff"
server_key_version: "encryptionkey@0"
I20260925 09:19:01.644558 24654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:01.666790 24654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:01.671269 24654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:01.671561 24654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:01.693972 24654 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41649
I20260925 09:19:01.694034 24720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41649 every 8 connection(s)
I20260925 09:19:01.696262 24654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:19:01.700856 24721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:01.706449 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 24654
I20260925 09:19:01.706636 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:19:01.707221 3149 external_mini_cluster.cc:1428] Setting key 4aa2bee491ea9761cd1ad6da592167db
I20260925 09:19:01.718602 24721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342: Bootstrap starting.
I20260925 09:19:01.723129 24721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342: Neither blocks nor log segments found. Creating new log.
Sep 25 09:19:01 dist-test-slave-l6qh krb5kdc[24641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 09:19:01.724632 24721 log.cc:824] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:01.728505 24721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342: No bootstrap required, opened a new log
I20260925 09:19:01.734216 24724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:01.709216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:58155 (local address 127.0.0.1:41649)
0925 09:19:01.710145 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:01.710164 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:01.710218 (+ 54us) server_negotiation.cc:407] Connection header received
0925 09:19:01.712960 (+ 2742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:01.712995 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:01.713637 (+ 642us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:01.714052 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:01.715944 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:01.717315 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:01.718623 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:01.719821 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:01.720027 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:01.725664 (+ 5637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:01.725680 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:01.725700 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:01.725748 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:01.729493 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:01.730497 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:01.730510 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:01.730517 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:01.730710 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:01.731250 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:01.731253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:01.731256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:01.731783 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:01.732129 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:01.732624 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 09:19:01.733153 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":353,"threads_started":1}
I20260925 09:19:01.738299 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:19:01.740018 24721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db81ff777e04798a309b26ac5fe2342" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41649 } }
I20260925 09:19:01.740402 24721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:19:01.740513 24721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db81ff777e04798a309b26ac5fe2342, State: Initialized, Role: FOLLOWER
I20260925 09:19:01.741151 24721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db81ff777e04798a309b26ac5fe2342" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41649 } }
I20260925 09:19:01.741463 24721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:19:01.741619 24721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:19:01.741806 24721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:01.743957 24721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db81ff777e04798a309b26ac5fe2342" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41649 } }
I20260925 09:19:01.744453 24721 leader_election.cc:302] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8db81ff777e04798a309b26ac5fe2342; no voters:
I20260925 09:19:01.745147 24721 leader_election.cc:288] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:19:01.745365 24730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:19:01.746791 24730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [term 1 LEADER]: Becoming Leader. State: Replica: 8db81ff777e04798a309b26ac5fe2342, State: Running, Role: LEADER
I20260925 09:19:01.747428 24730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db81ff777e04798a309b26ac5fe2342" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41649 } }
I20260925 09:19:01.747632 24721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:01.750665 24733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8db81ff777e04798a309b26ac5fe2342. Latest consensus state: current_term: 1 leader_uuid: "8db81ff777e04798a309b26ac5fe2342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db81ff777e04798a309b26ac5fe2342" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41649 } } }
I20260925 09:19:01.751132 24733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:01.751847 24731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8db81ff777e04798a309b26ac5fe2342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db81ff777e04798a309b26ac5fe2342" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41649 } } }
I20260925 09:19:01.752195 24731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:01.754537 24736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:19:01.761732 24736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:19:01.765908 24724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:01.745510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:57581 (local address 127.0.0.1:41649)
0925 09:19:01.745894 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:01.745900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:01.747776 (+ 1876us) server_negotiation.cc:407] Connection header received
0925 09:19:01.747972 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:01.747978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:01.748054 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:01.748168 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:01.750176 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:01.751436 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:01.753724 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:01.754601 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:01.754735 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:01.758104 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:01.758112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:01.758116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:01.758151 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:01.762072 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:01.763144 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:01.763150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:01.763154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:01.763295 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:01.763896 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:01.763901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:01.763905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:01.764350 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:01.764622 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:01.765329 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 09:19:01.765522 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260925 09:19:01.776546 24736 catalog_manager.cc:1347] Generated new cluster ID: a58561d29397432c87c264f6bebc1d24
I20260925 09:19:01.776675 24736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:19:01.814698 24736 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:19:01.816116 24736 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:19:01.833835 24736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8db81ff777e04798a309b26ac5fe2342: Generated new TSK 0
I20260925 09:19:01.834728 24736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:19:01.843284 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24654
I20260925 09:19:01.844437 24716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:19:01.940424 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 24654
W20260925 09:19:01.958441 24639 connection.cc:535] client connection to 127.0.0.1:41649 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41649): Connection reset by peer (error 104)
2026-09-25T09:19:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2070 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:19:02 dist-test-slave-l6qh krb5kdc[24754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:19:02 dist-test-slave-l6qh krb5kdc[24754](info): set up 2 sockets
Sep 25 09:19:02 dist-test-slave-l6qh krb5kdc[24754](info): commencing operation
krb5kdc: starting...
W20260925 09:19:03.038563 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:19:03 dist-test-slave-l6qh krb5kdc[24754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:19:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:19:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 09:19:03.297192 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:19:03.626502 24767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:03.626760 24767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:03.626849 24767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:03.633697 24767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:03.633795 24767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:03.633867 24767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:03.633919 24767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:03.633962 24767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:03.643230 24767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:03.644402 24767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:03.645637 24767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:19:03.660588 24772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:03.660745 24773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:03.661047 24775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:03.662084 24767 server_base.cc:1034] running on GCE node
I20260925 09:19:03.662900 24767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:03.664361 24767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:03.665625 24767 hybrid_clock.cc:648] HybridClock initialized: now 1790327943665585 us; error 32 us; skew 500 ppm
I20260925 09:19:03.666011 24767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:03 dist-test-slave-l6qh krb5kdc[24754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:03.672142 24767 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 09:19:03.673822 24767 webserver.cc:460] Webserver started at http://127.0.0.1:41145/ using document root <none> and password file <none>
I20260925 09:19:03.674510 24767 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:03.674655 24767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:03.674968 24767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:03.678153 24767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "2c12c1563de94a3d9e4ce35c113daa07"
format_stamp: "Formatted at 2026-09-25 09:19:03 on dist-test-slave-l6qh"
server_key: "8db3fada894baa613a94a5c45596f711"
server_key_iv: "5e630ca95f06838350cca238b57af7df"
server_key_version: "encryptionkey@0"
I20260925 09:19:03.679081 24767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "2c12c1563de94a3d9e4ce35c113daa07"
format_stamp: "Formatted at 2026-09-25 09:19:03 on dist-test-slave-l6qh"
server_key: "8db3fada894baa613a94a5c45596f711"
server_key_iv: "5e630ca95f06838350cca238b57af7df"
server_key_version: "encryptionkey@0"
I20260925 09:19:03.684616 24767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:19:03.688409 24782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:03.689800 24767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:19:03.690006 24767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "2c12c1563de94a3d9e4ce35c113daa07"
format_stamp: "Formatted at 2026-09-25 09:19:03 on dist-test-slave-l6qh"
server_key: "8db3fada894baa613a94a5c45596f711"
server_key_iv: "5e630ca95f06838350cca238b57af7df"
server_key_version: "encryptionkey@0"
I20260925 09:19:03.690202 24767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:03.710127 24767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:03.714025 24767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:03.714304 24767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:03.732748 24767 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43843
I20260925 09:19:03.732771 24833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43843 every 8 connection(s)
I20260925 09:19:03.734099 24767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:19:03.737709 24834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:03.743481 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 24767
I20260925 09:19:03.743667 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:19:03.744118 3149 external_mini_cluster.cc:1428] Setting key a799d0f0a361804b10be8fee7fbcdd3b
I20260925 09:19:03.749243 24834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07: Bootstrap starting.
I20260925 09:19:03.753088 24834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:03.754066 24834 log.cc:824] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:03.756616 24834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07: No bootstrap required, opened a new log
I20260925 09:19:03.758685 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:03.746062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:38017 (local address 127.0.0.1:43843)
0925 09:19:03.746899 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:03.746914 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:03.746967 (+ 53us) server_negotiation.cc:407] Connection header received
0925 09:19:03.749354 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:03.749381 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:03.749875 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:03.750271 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:03.750765 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:03.752797 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:03.754205 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:03.755072 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:03.755260 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:03.756461 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:03.756468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:03.756480 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:19:03.757037 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:03.757104 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:19:03.757129 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:03.757295 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:03.757627 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 09:19:03.757774 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":353,"threads_started":1}
I20260925 09:19:03.762773 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:19:03.766669 24834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c12c1563de94a3d9e4ce35c113daa07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } }
I20260925 09:19:03.767168 24834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:19:03.767285 24834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c12c1563de94a3d9e4ce35c113daa07, State: Initialized, Role: FOLLOWER
I20260925 09:19:03.767925 24834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c12c1563de94a3d9e4ce35c113daa07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } }
I20260925 09:19:03.768221 24834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:19:03.768371 24834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:19:03.768549 24834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:03.771701 24834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c12c1563de94a3d9e4ce35c113daa07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } }
I20260925 09:19:03.772276 24834 leader_election.cc:302] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c12c1563de94a3d9e4ce35c113daa07; no voters:
I20260925 09:19:03.772933 24834 leader_election.cc:288] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:19:03.773064 24844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:19:03.775105 24844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [term 1 LEADER]: Becoming Leader. State: Replica: 2c12c1563de94a3d9e4ce35c113daa07, State: Running, Role: LEADER
I20260925 09:19:03.775995 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:03.768026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:37749 (local address 127.0.0.1:43843)
0925 09:19:03.768210 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:03.768222 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:03.768792 (+ 570us) server_negotiation.cc:407] Connection header received
0925 09:19:03.768933 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:03.768946 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:03.769039 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:03.769148 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:03.769688 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:03.770866 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:03.772296 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:03.773108 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:03.773281 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:03.774860 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:03.774866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:03.774870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:19:03.774892 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:03.774964 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:19:03.774985 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:03.775125 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:03.775606 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 09:19:03.775663 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 09:19:03.775943 24844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c12c1563de94a3d9e4ce35c113daa07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } }
I20260925 09:19:03.776237 24834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:03.780220 24845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c12c1563de94a3d9e4ce35c113daa07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c12c1563de94a3d9e4ce35c113daa07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } } }
I20260925 09:19:03.780669 24845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:03.780740 24846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c12c1563de94a3d9e4ce35c113daa07. Latest consensus state: current_term: 1 leader_uuid: "2c12c1563de94a3d9e4ce35c113daa07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c12c1563de94a3d9e4ce35c113daa07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } } }
I20260925 09:19:03.781019 24846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:03.782627 24853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:19:03.787242 24853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:19:03.796613 24853 catalog_manager.cc:1347] Generated new cluster ID: 1d2c90942b734677ad72404c3a4416e7
I20260925 09:19:03.796737 24853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:19:03.813346 24853 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:19:03.814751 24853 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:19:03.821938 24853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c12c1563de94a3d9e4ce35c113daa07: Generated new TSK 0
I20260925 09:19:03.822984 24853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:19:03.828948 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24767
W20260925 09:19:03.830312 24829 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.32:38017
W20260925 09:19:03.831034 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24767: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:19:03.831148 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 24767
W20260925 09:19:03.848759 24752 connection.cc:535] client connection to 127.0.0.1:43843 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43843): Connection reset by peer (error 104)
2026-09-25T09:19:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1894 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:19:03 dist-test-slave-l6qh krb5kdc[24865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:19:03 dist-test-slave-l6qh krb5kdc[24865](info): set up 2 sockets
Sep 25 09:19:03 dist-test-slave-l6qh krb5kdc[24865](info): commencing operation
krb5kdc: starting...
W20260925 09:19:04.946725 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:19:05 dist-test-slave-l6qh krb5kdc[24865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:19:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:19:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 09:19:05.218597 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:19:05.547130 24878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:05.547477 24878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:05.547667 24878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:05.554888 24878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:05.555045 24878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:05.555131 24878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:05.555233 24878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:05.555307 24878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:05.565006 24878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:05.566351 24878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:05.567705 24878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:19:05.579957 24883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:05.580040 24884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:05.579910 24886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:05.580876 24878 server_base.cc:1034] running on GCE node
I20260925 09:19:05.581727 24878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:05.583024 24878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:05.584251 24878 hybrid_clock.cc:648] HybridClock initialized: now 1790327945584207 us; error 39 us; skew 500 ppm
I20260925 09:19:05.584594 24878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:05 dist-test-slave-l6qh krb5kdc[24865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:05.587641 24878 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 09:19:05.589102 24878 webserver.cc:460] Webserver started at http://127.0.0.1:42517/ using document root <none> and password file <none>
I20260925 09:19:05.589794 24878 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:05.589938 24878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:05.590260 24878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:05.593102 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "ef7b8cfa7f80449eb656945bddc42534"
format_stamp: "Formatted at 2026-09-25 09:19:05 on dist-test-slave-l6qh"
server_key: "7d0163a4e9d27e77ac838bd0c779aafb"
server_key_iv: "3726365cc4fbe67a75825ecf5cde6ae5"
server_key_version: "encryptionkey@0"
I20260925 09:19:05.593940 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "ef7b8cfa7f80449eb656945bddc42534"
format_stamp: "Formatted at 2026-09-25 09:19:05 on dist-test-slave-l6qh"
server_key: "7d0163a4e9d27e77ac838bd0c779aafb"
server_key_iv: "3726365cc4fbe67a75825ecf5cde6ae5"
server_key_version: "encryptionkey@0"
I20260925 09:19:05.598809 24878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 09:19:05.602169 24893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:05.603394 24878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 09:19:05.603565 24878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "ef7b8cfa7f80449eb656945bddc42534"
format_stamp: "Formatted at 2026-09-25 09:19:05 on dist-test-slave-l6qh"
server_key: "7d0163a4e9d27e77ac838bd0c779aafb"
server_key_iv: "3726365cc4fbe67a75825ecf5cde6ae5"
server_key_version: "encryptionkey@0"
I20260925 09:19:05.603771 24878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:05.609365 24878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:05.613044 24878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:05.613422 24878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:05.631518 24878 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43211
I20260925 09:19:05.631594 24944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43211 every 8 connection(s)
I20260925 09:19:05.632844 24878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:19:05.636348 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:05.642300 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 24878
I20260925 09:19:05.642493 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:19:05.643018 3149 external_mini_cluster.cc:1428] Setting key 572b498ec3f8545d86a9a1faed5380d1
I20260925 09:19:05.647571 24945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534: Bootstrap starting.
I20260925 09:19:05.652402 24945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:05.652355 24948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:05.644929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:45445 (local address 127.0.0.1:43211)
0925 09:19:05.645803 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:05.645814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:05.645854 (+ 40us) server_negotiation.cc:407] Connection header received
0925 09:19:05.647512 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:05.647535 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:05.647937 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:05.648410 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:05.649825 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:05.649834 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:05.649853 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:19:05.649895 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:05.650009 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:19:05.650038 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:05.650305 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:05.650449 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 09:19:05.650839 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":376,"threads_started":1}
I20260925 09:19:05.654114 24945 log.cc:824] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:05.657272 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:19:05.658059 24945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534: No bootstrap required, opened a new log
I20260925 09:19:05.664099 24948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:05.661590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:49361 (local address 127.0.0.1:43211)
0925 09:19:05.661797 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:05.661803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:05.662193 (+ 390us) server_negotiation.cc:407] Connection header received
0925 09:19:05.662331 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:05.662335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:05.662402 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:05.662527 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:05.663409 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:05.663416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:05.663421 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:19:05.663444 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:05.663495 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:19:05.663516 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:05.663641 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:05.663698 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 09:19:05.663756 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 09:19:05.667126 24945 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7b8cfa7f80449eb656945bddc42534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43211 } }
I20260925 09:19:05.667496 24945 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:19:05.667586 24945 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef7b8cfa7f80449eb656945bddc42534, State: Initialized, Role: FOLLOWER
I20260925 09:19:05.668164 24945 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7b8cfa7f80449eb656945bddc42534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43211 } }
I20260925 09:19:05.668432 24945 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:19:05.668594 24945 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:19:05.668777 24945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:05.671247 24945 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7b8cfa7f80449eb656945bddc42534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43211 } }
I20260925 09:19:05.671743 24945 leader_election.cc:302] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef7b8cfa7f80449eb656945bddc42534; no voters:
I20260925 09:19:05.672375 24945 leader_election.cc:288] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:19:05.672547 24955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:19:05.673277 24955 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [term 1 LEADER]: Becoming Leader. State: Replica: ef7b8cfa7f80449eb656945bddc42534, State: Running, Role: LEADER
I20260925 09:19:05.673808 24955 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7b8cfa7f80449eb656945bddc42534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43211 } }
I20260925 09:19:05.674757 24945 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:05.676862 24957 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef7b8cfa7f80449eb656945bddc42534. Latest consensus state: current_term: 1 leader_uuid: "ef7b8cfa7f80449eb656945bddc42534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7b8cfa7f80449eb656945bddc42534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43211 } } }
I20260925 09:19:05.677330 24957 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:05.677886 24956 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef7b8cfa7f80449eb656945bddc42534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7b8cfa7f80449eb656945bddc42534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43211 } } }
I20260925 09:19:05.678221 24956 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:05.678370 24964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:19:05.690227 24964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:19:05.699983 24964 catalog_manager.cc:1347] Generated new cluster ID: f69bb125f70d449394047d8155023958
I20260925 09:19:05.700119 24964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:19:05.715070 24964 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:19:05.715528 24964 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:19:05.723153 24964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef7b8cfa7f80449eb656945bddc42534: Generated new TSK 0
I20260925 09:19:05.723887 24964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:19:05.732962 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24878
W20260925 09:19:05.734311 24940 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.32:45445
W20260925 09:19:05.735013 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24878: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:19:05.735173 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 24878
2026-09-25T09:19:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1899 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:19:05 dist-test-slave-l6qh krb5kdc[24977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:19:05 dist-test-slave-l6qh krb5kdc[24977](info): set up 2 sockets
Sep 25 09:19:05 dist-test-slave-l6qh krb5kdc[24977](info): commencing operation
krb5kdc: starting...
W20260925 09:19:06.858721 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:19:07 dist-test-slave-l6qh krb5kdc[24977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:19:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:19:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:19:07.126223 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33155
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:35285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:33155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:19:07.457367 24990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:07.457664 24990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:07.457758 24990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:07.465073 24990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:07.465188 24990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:07.465281 24990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:07.465366 24990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:07.465405 24990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:07.475610 24990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:33155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:33155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:07.476854 24990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:07.478216 24990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:19:07.490688 24996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:07.490774 24998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:07.490731 24995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:07.491701 24990 server_base.cc:1034] running on GCE node
I20260925 09:19:07.492570 24990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:07.494057 24990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:07.495260 24990 hybrid_clock.cc:648] HybridClock initialized: now 1790327947495225 us; error 30 us; skew 500 ppm
I20260925 09:19:07.495576 24990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:07 dist-test-slave-l6qh krb5kdc[24977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:07.498631 24990 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:19:07.500066 24990 webserver.cc:460] Webserver started at http://127.3.19.126:42043/ using document root <none> and password file <none>
I20260925 09:19:07.500792 24990 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:07.500924 24990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:07.501391 24990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:07.504386 24990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "fd64f172aa3b4497a12724a890d020f8"
format_stamp: "Formatted at 2026-09-25 09:19:07 on dist-test-slave-l6qh"
server_key: "1a327ce359c4e597b674252f07ecf8b3"
server_key_iv: "47471dacc082da92c67e6b76b2eb64e3"
server_key_version: "encryptionkey@0"
I20260925 09:19:07.505172 24990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "fd64f172aa3b4497a12724a890d020f8"
format_stamp: "Formatted at 2026-09-25 09:19:07 on dist-test-slave-l6qh"
server_key: "1a327ce359c4e597b674252f07ecf8b3"
server_key_iv: "47471dacc082da92c67e6b76b2eb64e3"
server_key_version: "encryptionkey@0"
I20260925 09:19:07.510399 24990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 09:19:07.513720 25005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:07.514981 24990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 09:19:07.515167 24990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "fd64f172aa3b4497a12724a890d020f8"
format_stamp: "Formatted at 2026-09-25 09:19:07 on dist-test-slave-l6qh"
server_key: "1a327ce359c4e597b674252f07ecf8b3"
server_key_iv: "47471dacc082da92c67e6b76b2eb64e3"
server_key_version: "encryptionkey@0"
I20260925 09:19:07.515350 24990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:07.528198 24990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:07.532377 24990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:07.532681 24990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:07.551339 24990 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:33155
I20260925 09:19:07.551371 25056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:33155 every 8 connection(s)
I20260925 09:19:07.552716 24990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:19:07.556393 25057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:07.560596 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 24990
I20260925 09:19:07.560781 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:19:07.561373 3149 external_mini_cluster.cc:1428] Setting key 301856c973eecfbd9c5e0f052dc6d299
I20260925 09:19:07.570449 25057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8: Bootstrap starting.
I20260925 09:19:07.574379 25057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:07.575721 25057 log.cc:824] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8: Log is configured to *not* fsync() on all Append() calls
Sep 25 09:19:07 dist-test-slave-l6qh krb5kdc[24977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:19:07.578794 25057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8: No bootstrap required, opened a new log
I20260925 09:19:07.585220 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:07.563291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53955 (local address 127.3.19.126:33155)
0925 09:19:07.564396 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:07.564408 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:07.564453 (+ 45us) server_negotiation.cc:407] Connection header received
0925 09:19:07.566123 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:07.566146 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:07.566556 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:07.566915 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:07.568710 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:07.570104 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:07.571797 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:07.572482 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:07.572635 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:07.577921 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:07.577929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:07.577942 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:07.577978 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:07.581120 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:07.582008 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:07.582014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:07.582020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:07.582183 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:07.582664 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:07.582667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:07.582669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:07.583144 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:07.583393 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:07.583890 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 09:19:07.584204 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":661,"threads_started":1}
I20260925 09:19:07.587400 25057 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd64f172aa3b4497a12724a890d020f8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33155 } }
I20260925 09:19:07.587782 25057 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:19:07.587903 25057 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd64f172aa3b4497a12724a890d020f8, State: Initialized, Role: FOLLOWER
I20260925 09:19:07.588511 25057 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd64f172aa3b4497a12724a890d020f8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33155 } }
I20260925 09:19:07.588768 25057 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:19:07.588917 25057 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:19:07.589097 25057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:07.589161 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:19:07.592231 25057 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd64f172aa3b4497a12724a890d020f8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33155 } }
I20260925 09:19:07.592880 25057 leader_election.cc:302] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd64f172aa3b4497a12724a890d020f8; no voters:
I20260925 09:19:07.593571 25057 leader_election.cc:288] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:19:07.593633 25066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:19:07.596064 25057 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:07.598531 25066 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [term 1 LEADER]: Becoming Leader. State: Replica: fd64f172aa3b4497a12724a890d020f8, State: Running, Role: LEADER
I20260925 09:19:07.599282 25066 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd64f172aa3b4497a12724a890d020f8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33155 } }
I20260925 09:19:07.602378 25068 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd64f172aa3b4497a12724a890d020f8. Latest consensus state: current_term: 1 leader_uuid: "fd64f172aa3b4497a12724a890d020f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd64f172aa3b4497a12724a890d020f8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33155 } } }
I20260925 09:19:07.602818 25068 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:07.602959 25067 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd64f172aa3b4497a12724a890d020f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd64f172aa3b4497a12724a890d020f8" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 33155 } } }
I20260925 09:19:07.603300 25067 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:07.604243 25078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:19:07.614843 25078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:19:07.618939 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:07.594763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44683 (local address 127.3.19.126:33155)
0925 09:19:07.594983 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:07.594992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:07.597989 (+ 2997us) server_negotiation.cc:407] Connection header received
0925 09:19:07.598321 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:07.598330 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:07.598434 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:07.598520 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:07.600724 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:07.602471 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:07.604451 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:07.605114 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:07.605321 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:07.609500 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:07.609508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:07.609512 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:07.609548 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:07.614250 (+ 4702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:07.615528 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:07.615537 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:07.615544 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:07.615749 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:07.616420 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:07.616424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:07.616427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:07.616853 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:07.617135 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:07.618305 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0925 09:19:07.618528 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 09:19:07.626590 25078 catalog_manager.cc:1347] Generated new cluster ID: 02a178a8bfbb4fcabf3d20eb3dde4ddc
I20260925 09:19:07.626706 25078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:19:07.640130 25078 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:19:07.641842 25078 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:19:07.660296 25078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd64f172aa3b4497a12724a890d020f8: Generated new TSK 0
I20260925 09:19:07.661379 25078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:19:07.674666 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 24990
I20260925 09:19:07.675835 25052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:19:07.768306 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 24990
W20260925 09:19:07.786778 24975 connection.cc:535] client connection to 127.3.19.126:33155 recv error: Network error: failed to read from TLS socket (remote: 127.3.19.126:33155): Connection reset by peer (error 104)
2026-09-25T09:19:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:19:07 dist-test-slave-l6qh krb5kdc[25089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:19:07 dist-test-slave-l6qh krb5kdc[25089](info): set up 2 sockets
Sep 25 09:19:07 dist-test-slave-l6qh krb5kdc[25089](info): commencing operation
krb5kdc: starting...
W20260925 09:19:08.882692 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:19:09 dist-test-slave-l6qh krb5kdc[25089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:19:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:19:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:19:09.160681 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:43381
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:43381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:19:09.493191 25102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:09.493543 25102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:09.493698 25102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:09.500784 25102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:09.500936 25102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:09.501030 25102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:09.501143 25102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:09.501226 25102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:09.510891 25102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:43381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:43381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:09.512181 25102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:09.513568 25102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:19:09.525624 25107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:09.525667 25108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:09.525694 25110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:09.526379 25102 server_base.cc:1034] running on GCE node
I20260925 09:19:09.527073 25102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:09.528347 25102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:09.529539 25102 hybrid_clock.cc:648] HybridClock initialized: now 1790327949529495 us; error 44 us; skew 500 ppm
I20260925 09:19:09.529891 25102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:09 dist-test-slave-l6qh krb5kdc[25089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327949, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:09.539386 25102 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:19:09.540897 25102 webserver.cc:460] Webserver started at http://127.3.19.126:44881/ using document root <none> and password file <none>
I20260925 09:19:09.541638 25102 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:09.541766 25102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:09.542092 25102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:09.545028 25102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "0799abbed2014fc1921996dd5df75b1c"
format_stamp: "Formatted at 2026-09-25 09:19:09 on dist-test-slave-l6qh"
server_key: "d91a7eb19456b08cfb10ce95e1853f61"
server_key_iv: "5785b8cd1eaa5b3babde4fc8c7e64ee9"
server_key_version: "encryptionkey@0"
I20260925 09:19:09.545841 25102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "0799abbed2014fc1921996dd5df75b1c"
format_stamp: "Formatted at 2026-09-25 09:19:09 on dist-test-slave-l6qh"
server_key: "d91a7eb19456b08cfb10ce95e1853f61"
server_key_iv: "5785b8cd1eaa5b3babde4fc8c7e64ee9"
server_key_version: "encryptionkey@0"
I20260925 09:19:09.550786 25102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 09:19:09.554190 25117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:09.555469 25102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 09:19:09.555653 25102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "0799abbed2014fc1921996dd5df75b1c"
format_stamp: "Formatted at 2026-09-25 09:19:09 on dist-test-slave-l6qh"
server_key: "d91a7eb19456b08cfb10ce95e1853f61"
server_key_iv: "5785b8cd1eaa5b3babde4fc8c7e64ee9"
server_key_version: "encryptionkey@0"
I20260925 09:19:09.555841 25102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:09.576283 25102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:09.580479 25102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:09.580777 25102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:09.599839 25102 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:43381
I20260925 09:19:09.599872 25168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:43381 every 8 connection(s)
I20260925 09:19:09.601212 25102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:19:09.604856 25169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:09.607589 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 25102
I20260925 09:19:09.607774 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:19:09.608326 3149 external_mini_cluster.cc:1428] Setting key f330549bbe7c9aa6d13ae4bfcbaf154b
I20260925 09:19:09.616801 25169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c: Bootstrap starting.
I20260925 09:19:09.620404 25169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:09.621441 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:09.610089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32955 (local address 127.3.19.126:43381)
0925 09:19:09.610917 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:09.610928 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:09.610968 (+ 40us) server_negotiation.cc:407] Connection header received
0925 09:19:09.612553 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:09.612574 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:09.612955 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:09.613330 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:09.613874 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:09.615220 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:09.617047 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:09.617740 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:09.617865 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:09.619162 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:09.619170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:09.619182 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:19:09.619703 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:09.619779 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:19:09.619804 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:09.619919 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:09.620202 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 09:19:09.620401 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":391,"threads_started":1}
I20260925 09:19:09.621517 25169 log.cc:824] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:09.624675 25169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c: No bootstrap required, opened a new log
I20260925 09:19:09.625427 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:19:09.632987 25169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0799abbed2014fc1921996dd5df75b1c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43381 } }
I20260925 09:19:09.633417 25169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:19:09.633550 25169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0799abbed2014fc1921996dd5df75b1c, State: Initialized, Role: FOLLOWER
I20260925 09:19:09.634222 25169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0799abbed2014fc1921996dd5df75b1c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43381 } }
I20260925 09:19:09.634516 25169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:19:09.634678 25169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:19:09.634865 25169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:09.636170 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:09.629590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37459 (local address 127.3.19.126:43381)
0925 09:19:09.629771 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:09.629781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:09.630197 (+ 416us) server_negotiation.cc:407] Connection header received
0925 09:19:09.630334 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:09.630339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:09.630406 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:09.630513 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:09.630964 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:09.632146 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:09.633502 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:09.634165 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:09.634305 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:09.635416 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:09.635422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:09.635426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:19:09.635447 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:09.635495 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:19:09.635519 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:09.635659 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:09.635783 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 09:19:09.635836 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 09:19:09.637704 25169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0799abbed2014fc1921996dd5df75b1c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43381 } }
I20260925 09:19:09.638181 25169 leader_election.cc:302] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0799abbed2014fc1921996dd5df75b1c; no voters:
I20260925 09:19:09.638729 25169 leader_election.cc:288] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:19:09.638888 25179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:19:09.639592 25179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [term 1 LEADER]: Becoming Leader. State: Replica: 0799abbed2014fc1921996dd5df75b1c, State: Running, Role: LEADER
I20260925 09:19:09.640159 25179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0799abbed2014fc1921996dd5df75b1c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43381 } }
I20260925 09:19:09.640949 25169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:09.643227 25181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0799abbed2014fc1921996dd5df75b1c. Latest consensus state: current_term: 1 leader_uuid: "0799abbed2014fc1921996dd5df75b1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0799abbed2014fc1921996dd5df75b1c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43381 } } }
I20260925 09:19:09.643718 25181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:09.644279 25180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0799abbed2014fc1921996dd5df75b1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0799abbed2014fc1921996dd5df75b1c" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43381 } } }
I20260925 09:19:09.644625 25180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:09.646279 25187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:19:09.667281 25187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:19:09.677524 25187 catalog_manager.cc:1347] Generated new cluster ID: 0b9aabcba26744b192f7b294a5e6721d
I20260925 09:19:09.677644 25187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:19:09.692103 25187 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:19:09.693908 25187 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:19:09.708281 25187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0799abbed2014fc1921996dd5df75b1c: Generated new TSK 0
I20260925 09:19:09.709087 25187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:19:09.716490 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 25102
W20260925 09:19:09.717659 25164 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32955
W20260925 09:19:09.718350 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 25102: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:19:09.718480 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 25102
W20260925 09:19:09.734387 25087 connection.cc:535] client connection to 127.3.19.126:43381 recv error: Network error: failed to read from TLS socket (remote: 127.3.19.126:43381): Connection reset by peer (error 104)
2026-09-25T09:19:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:19:09 dist-test-slave-l6qh krb5kdc[25200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:19:09 dist-test-slave-l6qh krb5kdc[25200](info): set up 2 sockets
Sep 25 09:19:09 dist-test-slave-l6qh krb5kdc[25200](info): commencing operation
krb5kdc: starting...
W20260925 09:19:10.815127 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:19:10 dist-test-slave-l6qh krb5kdc[25200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:19:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:19:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:19:11.086453 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:37761
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:37761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:19:11.416803 25213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:11.417156 25213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:11.417378 25213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:11.424770 25213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:11.424932 25213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:11.425050 25213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:11.425165 25213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:11.425256 25213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:11.435328 25213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:37761
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:37761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:11.436976 25213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:11.438458 25213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:19:11.450906 25218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:11.450999 25219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:11.451179 25221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:11.451903 25213 server_base.cc:1034] running on GCE node
I20260925 09:19:11.452715 25213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:11.454257 25213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:11.455469 25213 hybrid_clock.cc:648] HybridClock initialized: now 1790327951455431 us; error 38 us; skew 500 ppm
I20260925 09:19:11.455811 25213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:11 dist-test-slave-l6qh krb5kdc[25200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327951, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:11.459250 25213 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:19:11.460861 25213 webserver.cc:460] Webserver started at http://127.3.19.126:36523/ using document root <none> and password file <none>
I20260925 09:19:11.461680 25213 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:11.461795 25213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:11.462070 25213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:11.465323 25213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "06fb35fd49774bb8b86ad3ed3ee9becb"
format_stamp: "Formatted at 2026-09-25 09:19:11 on dist-test-slave-l6qh"
server_key: "66219fc018b8158c785bc727ccf4c0e6"
server_key_iv: "b3eeb188b37b4511d4712a36d46cbddf"
server_key_version: "encryptionkey@0"
I20260925 09:19:11.466162 25213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "06fb35fd49774bb8b86ad3ed3ee9becb"
format_stamp: "Formatted at 2026-09-25 09:19:11 on dist-test-slave-l6qh"
server_key: "66219fc018b8158c785bc727ccf4c0e6"
server_key_iv: "b3eeb188b37b4511d4712a36d46cbddf"
server_key_version: "encryptionkey@0"
I20260925 09:19:11.471726 25213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:19:11.475342 25228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:11.476719 25213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 09:19:11.476898 25213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "06fb35fd49774bb8b86ad3ed3ee9becb"
format_stamp: "Formatted at 2026-09-25 09:19:11 on dist-test-slave-l6qh"
server_key: "66219fc018b8158c785bc727ccf4c0e6"
server_key_iv: "b3eeb188b37b4511d4712a36d46cbddf"
server_key_version: "encryptionkey@0"
I20260925 09:19:11.477099 25213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:11.483062 25213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:11.487303 25213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:11.487615 25213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:11.506690 25213 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:37761
I20260925 09:19:11.506742 25279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:37761 every 8 connection(s)
I20260925 09:19:11.508157 25213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:19:11.512006 25280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:11.512217 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 25213
I20260925 09:19:11.512403 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:19:11.512943 3149 external_mini_cluster.cc:1428] Setting key 4c0bb5ea32923fa65271ed0de6deeacc
I20260925 09:19:11.520823 25283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:11.514790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49955 (local address 127.3.19.126:37761)
0925 09:19:11.515714 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:11.515731 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:11.515772 (+ 41us) server_negotiation.cc:407] Connection header received
0925 09:19:11.517363 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:11.517385 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:11.517732 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:11.518090 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:11.519182 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:11.519188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:11.519201 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:19:11.519234 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:11.519308 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:19:11.519331 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:11.519555 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:11.519672 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 09:19:11.519900 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":435,"threads_started":1}
I20260925 09:19:11.524639 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:19:11.526249 25280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb: Bootstrap starting.
I20260925 09:19:11.529572 25280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:11.530790 25280 log.cc:824] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:11.531909 25283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:11.529080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35011 (local address 127.3.19.126:37761)
0925 09:19:11.529287 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:11.529293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:11.529800 (+ 507us) server_negotiation.cc:407] Connection header received
0925 09:19:11.529942 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:11.529947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:11.530025 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:11.530143 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:11.531125 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:11.531150 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:11.531155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:19:11.531173 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:11.531222 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:19:11.531247 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:11.531399 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:11.531549 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 09:19:11.531600 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 09:19:11.533913 25280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb: No bootstrap required, opened a new log
I20260925 09:19:11.541618 25280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fb35fd49774bb8b86ad3ed3ee9becb" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37761 } }
I20260925 09:19:11.542029 25280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:19:11.542143 25280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06fb35fd49774bb8b86ad3ed3ee9becb, State: Initialized, Role: FOLLOWER
I20260925 09:19:11.542729 25280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fb35fd49774bb8b86ad3ed3ee9becb" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37761 } }
I20260925 09:19:11.542979 25280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:19:11.543129 25280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:19:11.543298 25280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:11.545637 25280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fb35fd49774bb8b86ad3ed3ee9becb" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37761 } }
I20260925 09:19:11.546171 25280 leader_election.cc:302] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06fb35fd49774bb8b86ad3ed3ee9becb; no voters:
I20260925 09:19:11.546761 25280 leader_election.cc:288] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:19:11.546912 25290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:19:11.547616 25290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [term 1 LEADER]: Becoming Leader. State: Replica: 06fb35fd49774bb8b86ad3ed3ee9becb, State: Running, Role: LEADER
I20260925 09:19:11.548156 25290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fb35fd49774bb8b86ad3ed3ee9becb" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37761 } }
I20260925 09:19:11.548878 25280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:11.551731 25291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06fb35fd49774bb8b86ad3ed3ee9becb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fb35fd49774bb8b86ad3ed3ee9becb" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37761 } } }
I20260925 09:19:11.551783 25292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06fb35fd49774bb8b86ad3ed3ee9becb. Latest consensus state: current_term: 1 leader_uuid: "06fb35fd49774bb8b86ad3ed3ee9becb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fb35fd49774bb8b86ad3ed3ee9becb" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 37761 } } }
I20260925 09:19:11.552302 25291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:11.552307 25292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:11.554829 25296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:19:11.560035 25296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:19:11.568622 25296 catalog_manager.cc:1347] Generated new cluster ID: cb8f5e4edd404bbabb1d4f55b50d9896
I20260925 09:19:11.568742 25296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:19:11.610510 25296 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:19:11.610967 25296 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:19:11.627380 25296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06fb35fd49774bb8b86ad3ed3ee9becb: Generated new TSK 0
I20260925 09:19:11.628283 25296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:19:11.636302 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 25213
W20260925 09:19:11.637501 25275 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49955
W20260925 09:19:11.638370 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 25213: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:19:11.638491 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 25213
2026-09-25T09:19:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1924 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:19:11 dist-test-slave-l6qh krb5kdc[25311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:19:11 dist-test-slave-l6qh krb5kdc[25311](info): set up 2 sockets
Sep 25 09:19:11 dist-test-slave-l6qh krb5kdc[25311](info): commencing operation
krb5kdc: starting...
W20260925 09:19:12.855300 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:19:13 dist-test-slave-l6qh krb5kdc[25311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:19:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:19:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:19:13.130504 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:43483
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:43483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:19:13.462839 25324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:13.463119 25324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:13.463299 25324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:13.470307 25324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:13.470402 25324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:13.470546 25324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:13.470654 25324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:13.470732 25324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:13.480136 25324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:43483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:43483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:13.481395 25324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:13.482713 25324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:19:13.494938 25329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:13.494967 25330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:13.495498 25332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:13.496035 25324 server_base.cc:1034] running on GCE node
I20260925 09:19:13.496826 25324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:13.498214 25324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:13.499408 25324 hybrid_clock.cc:648] HybridClock initialized: now 1790327953499343 us; error 63 us; skew 500 ppm
I20260925 09:19:13.499773 25324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:13 dist-test-slave-l6qh krb5kdc[25311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327953, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:13.502873 25324 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:19:13.504256 25324 webserver.cc:460] Webserver started at http://127.3.19.126:36515/ using document root <none> and password file <none>
I20260925 09:19:13.504918 25324 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:13.505070 25324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:13.505455 25324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:13.508343 25324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "106f50f1939a43fa983153aa54656463"
format_stamp: "Formatted at 2026-09-25 09:19:13 on dist-test-slave-l6qh"
server_key: "bbc15b2439d113ed64c6c6950b2595c3"
server_key_iv: "27c2815a83c7d58196755344879ad427"
server_key_version: "encryptionkey@0"
I20260925 09:19:13.509104 25324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "106f50f1939a43fa983153aa54656463"
format_stamp: "Formatted at 2026-09-25 09:19:13 on dist-test-slave-l6qh"
server_key: "bbc15b2439d113ed64c6c6950b2595c3"
server_key_iv: "27c2815a83c7d58196755344879ad427"
server_key_version: "encryptionkey@0"
I20260925 09:19:13.514143 25324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 09:19:13.517585 25339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:13.518869 25324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 09:19:13.519054 25324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "106f50f1939a43fa983153aa54656463"
format_stamp: "Formatted at 2026-09-25 09:19:13 on dist-test-slave-l6qh"
server_key: "bbc15b2439d113ed64c6c6950b2595c3"
server_key_iv: "27c2815a83c7d58196755344879ad427"
server_key_version: "encryptionkey@0"
I20260925 09:19:13.519269 25324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:13.556452 25324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:13.560750 25324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:13.561043 25324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:13.580295 25324 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:43483
I20260925 09:19:13.580350 25390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:43483 every 8 connection(s)
I20260925 09:19:13.581679 25324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:19:13.585302 25391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:13.586974 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 25324
I20260925 09:19:13.587160 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:19:13.587618 3149 external_mini_cluster.cc:1428] Setting key 91eb710e13fb39c74eececbf210fbfe9
I20260925 09:19:13.600236 25391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463: Bootstrap starting.
Sep 25 09:19:13 dist-test-slave-l6qh krb5kdc[25311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:19:13.603966 25391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:13.605319 25391 log.cc:824] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:13.608771 25391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463: No bootstrap required, opened a new log
I20260925 09:19:13.612389 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:13.589523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58845 (local address 127.3.19.126:43483)
0925 09:19:13.590349 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:13.590364 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:13.590420 (+ 56us) server_negotiation.cc:407] Connection header received
0925 09:19:13.592162 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:13.592189 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:13.592734 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:13.593185 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:13.594493 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:13.596554 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:13.597990 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:13.599085 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:13.599278 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:13.604188 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:13.604199 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:13.604216 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:13.604254 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:13.607893 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:13.608734 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:13.608739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:13.608746 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:13.608962 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:13.609400 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:13.609403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:13.609405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:13.609941 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:13.610166 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:13.610688 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 09:19:13.611092 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":337,"threads_started":1}
I20260925 09:19:13.616601 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:19:13.617760 25391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106f50f1939a43fa983153aa54656463" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43483 } }
I20260925 09:19:13.618140 25391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:19:13.618247 25391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106f50f1939a43fa983153aa54656463, State: Initialized, Role: FOLLOWER
I20260925 09:19:13.618819 25391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106f50f1939a43fa983153aa54656463" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43483 } }
I20260925 09:19:13.619072 25391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:19:13.619235 25391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:19:13.619406 25391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:13.621707 25391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106f50f1939a43fa983153aa54656463" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43483 } }
I20260925 09:19:13.622283 25391 leader_election.cc:302] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 106f50f1939a43fa983153aa54656463; no voters:
I20260925 09:19:13.622999 25391 leader_election.cc:288] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:19:13.623121 25400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:19:13.624748 25400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [term 1 LEADER]: Becoming Leader. State: Replica: 106f50f1939a43fa983153aa54656463, State: Running, Role: LEADER
I20260925 09:19:13.625386 25400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106f50f1939a43fa983153aa54656463" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43483 } }
I20260925 09:19:13.625965 25391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:13.628619 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 106f50f1939a43fa983153aa54656463. Latest consensus state: current_term: 1 leader_uuid: "106f50f1939a43fa983153aa54656463" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106f50f1939a43fa983153aa54656463" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43483 } } }
I20260925 09:19:13.629127 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:13.628588 25401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "106f50f1939a43fa983153aa54656463" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106f50f1939a43fa983153aa54656463" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43483 } } }
I20260925 09:19:13.630065 25401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:13.630282 25408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:19:13.645018 25408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:19:13.651818 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:13.623555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42213 (local address 127.3.19.126:43483)
0925 09:19:13.623787 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:13.623796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:13.630019 (+ 6223us) server_negotiation.cc:407] Connection header received
0925 09:19:13.630344 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:13.630353 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:13.630463 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:13.630560 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:13.633191 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:13.634689 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:13.637043 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:13.638013 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:13.638146 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:13.642494 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:13.642504 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:13.642511 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:13.642575 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:13.647597 (+ 5022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:13.648745 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:13.648751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:13.648756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:13.648925 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:13.649469 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:13.649478 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:13.649483 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:13.650061 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:13.650342 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:13.651048 (+ 706us) server_negotiation.cc:299] Negotiation successful
0925 09:19:13.651321 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 09:19:13.662289 25408 catalog_manager.cc:1347] Generated new cluster ID: bda24c76022e47d9a23febab95f0e574
I20260925 09:19:13.662410 25408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:19:13.675056 25408 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:19:13.676386 25408 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:19:13.689347 25408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 106f50f1939a43fa983153aa54656463: Generated new TSK 0
I20260925 09:19:13.690420 25408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:19:13.702172 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 25324
I20260925 09:19:13.703248 25386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:19:13.794715 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 25324
2026-09-25T09:19:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2153 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:19:13 dist-test-slave-l6qh krb5kdc[25423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:19:13 dist-test-slave-l6qh krb5kdc[25423](info): set up 2 sockets
Sep 25 09:19:13 dist-test-slave-l6qh krb5kdc[25423](info): commencing operation
krb5kdc: starting...
W20260925 09:19:14.898645 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:19:15 dist-test-slave-l6qh krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:19:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:19:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:19:15.168002 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:41127
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:43679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:41127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:19:15.497215 25436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:15.497566 25436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:15.497809 25436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:15.505045 25436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:15.505153 25436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:15.505249 25436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:15.505398 25436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:15.505457 25436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:15.515153 25436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:43679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:41127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:15.516475 25436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:15.517840 25436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:19:15.529173 25441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:15.529206 25442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:15.529268 25444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:15.530165 25436 server_base.cc:1034] running on GCE node
I20260925 09:19:15.530993 25436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:15.532472 25436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:15.533699 25436 hybrid_clock.cc:648] HybridClock initialized: now 1790327955533641 us; error 56 us; skew 500 ppm
I20260925 09:19:15.534034 25436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:15 dist-test-slave-l6qh krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327955, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:15.541019 25436 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:19:15.542738 25436 webserver.cc:460] Webserver started at http://127.3.19.126:38087/ using document root <none> and password file <none>
I20260925 09:19:15.543421 25436 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:15.543584 25436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:15.543893 25436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:15.546892 25436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "796ad3e9821346e68501df09746ada17"
format_stamp: "Formatted at 2026-09-25 09:19:15 on dist-test-slave-l6qh"
server_key: "70007aa3fef1952802c854fd377fd8d1"
server_key_iv: "73d67e4ad3817196ae9a3029b71a9632"
server_key_version: "encryptionkey@0"
I20260925 09:19:15.547668 25436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "796ad3e9821346e68501df09746ada17"
format_stamp: "Formatted at 2026-09-25 09:19:15 on dist-test-slave-l6qh"
server_key: "70007aa3fef1952802c854fd377fd8d1"
server_key_iv: "73d67e4ad3817196ae9a3029b71a9632"
server_key_version: "encryptionkey@0"
I20260925 09:19:15.552990 25436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 09:19:15.556550 25451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:15.557976 25436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:19:15.558174 25436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "796ad3e9821346e68501df09746ada17"
format_stamp: "Formatted at 2026-09-25 09:19:15 on dist-test-slave-l6qh"
server_key: "70007aa3fef1952802c854fd377fd8d1"
server_key_iv: "73d67e4ad3817196ae9a3029b71a9632"
server_key_version: "encryptionkey@0"
I20260925 09:19:15.558405 25436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:15.579639 25436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:15.583926 25436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:15.584221 25436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:15.604149 25436 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:41127
I20260925 09:19:15.604185 25502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:41127 every 8 connection(s)
I20260925 09:19:15.605711 25436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:19:15.609562 25503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:15.614034 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 25436
I20260925 09:19:15.614225 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:19:15.614765 3149 external_mini_cluster.cc:1428] Setting key 5a2a5089d4dbbf0228e27ed71d55f2fb
I20260925 09:19:15.620821 25503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17: Bootstrap starting.
I20260925 09:19:15.624360 25503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:15.625749 25503 log.cc:824] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:15.628711 25503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17: No bootstrap required, opened a new log
I20260925 09:19:15.629314 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:15.616552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56249 (local address 127.3.19.126:41127)
0925 09:19:15.617447 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:15.617459 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:15.617516 (+ 57us) server_negotiation.cc:407] Connection header received
0925 09:19:15.619570 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:15.619593 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:15.619985 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:15.620330 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:15.621000 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:15.622396 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:15.624192 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:15.625100 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:15.625303 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:15.626915 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:15.626923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:15.626941 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:19:15.627512 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:15.627589 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:19:15.627614 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:15.627750 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:15.627977 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 09:19:15.628230 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":408,"threads_started":1}
I20260925 09:19:15.632985 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:19:15.636742 25503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796ad3e9821346e68501df09746ada17" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 41127 } }
I20260925 09:19:15.637142 25503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:19:15.637302 25503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796ad3e9821346e68501df09746ada17, State: Initialized, Role: FOLLOWER
I20260925 09:19:15.638036 25503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796ad3e9821346e68501df09746ada17" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 41127 } }
I20260925 09:19:15.638310 25503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:19:15.638471 25503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:19:15.638685 25503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:15.641669 25503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796ad3e9821346e68501df09746ada17" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 41127 } }
I20260925 09:19:15.642220 25503 leader_election.cc:302] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 796ad3e9821346e68501df09746ada17; no voters:
I20260925 09:19:15.642999 25503 leader_election.cc:288] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:19:15.643103 25513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:19:15.644713 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:15.637763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52227 (local address 127.3.19.126:41127)
0925 09:19:15.637984 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:15.637992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:15.638263 (+ 271us) server_negotiation.cc:407] Connection header received
0925 09:19:15.638477 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:15.638483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:15.638560 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:15.638684 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:15.639332 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:15.640152 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:15.642013 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:15.642588 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:15.642742 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:15.644025 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:15.644030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:15.644033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:19:15.644050 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:15.644088 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:19:15.644103 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:15.644206 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:15.644378 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 09:19:15.644473 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 09:19:15.645506 25503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:15.648690 25513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [term 1 LEADER]: Becoming Leader. State: Replica: 796ad3e9821346e68501df09746ada17, State: Running, Role: LEADER
I20260925 09:19:15.649539 25513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796ad3e9821346e68501df09746ada17" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 41127 } }
I20260925 09:19:15.652599 25514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "796ad3e9821346e68501df09746ada17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796ad3e9821346e68501df09746ada17" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 41127 } } }
I20260925 09:19:15.652949 25514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:15.653442 25515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 796ad3e9821346e68501df09746ada17. Latest consensus state: current_term: 1 leader_uuid: "796ad3e9821346e68501df09746ada17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796ad3e9821346e68501df09746ada17" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 41127 } } }
I20260925 09:19:15.653748 25515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:15.658217 25522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:19:15.665083 25522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:19:15.674180 25522 catalog_manager.cc:1347] Generated new cluster ID: 78a482ab6e8e4c1898076ca00a37d1d4
I20260925 09:19:15.674321 25522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:19:15.719861 25522 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:19:15.721786 25522 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:19:15.730206 25522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 796ad3e9821346e68501df09746ada17: Generated new TSK 0
I20260925 09:19:15.731253 25522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:19:15.748693 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 25436
W20260925 09:19:15.749878 25498 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56249
W20260925 09:19:15.750589 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 25436: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:19:15.750725 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 25436
2026-09-25T09:19:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1953 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:19:15 dist-test-slave-l6qh krb5kdc[25534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:19:15 dist-test-slave-l6qh krb5kdc[25534](info): set up 2 sockets
Sep 25 09:19:15 dist-test-slave-l6qh krb5kdc[25534](info): commencing operation
krb5kdc: starting...
W20260925 09:19:16.866847 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:19:17 dist-test-slave-l6qh krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:19:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:19:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:19:17.161391 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:34725
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:34725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:19:17.498145 25547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:17.498425 25547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:17.498620 25547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:17.505745 25547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:17.505883 25547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:17.505980 25547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:17.506085 25547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:17.506168 25547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:17.515827 25547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:34725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:34725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:17.517112 25547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:17.518427 25547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:19:17.530599 25555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:17.530709 25553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:17.530714 25552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:17.531831 25547 server_base.cc:1034] running on GCE node
I20260925 09:19:17.532603 25547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:17.534102 25547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:17.535315 25547 hybrid_clock.cc:648] HybridClock initialized: now 1790327957535251 us; error 56 us; skew 500 ppm
I20260925 09:19:17.535707 25547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:17 dist-test-slave-l6qh krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327957, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:17.539108 25547 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:19:17.540689 25547 webserver.cc:460] Webserver started at http://127.3.19.126:44073/ using document root <none> and password file <none>
I20260925 09:19:17.541430 25547 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:17.541540 25547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:17.541810 25547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:17.544873 25547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "d818fcc6fc044378a8f6a662e3d22962"
format_stamp: "Formatted at 2026-09-25 09:19:17 on dist-test-slave-l6qh"
server_key: "ff7b3df6bf54753892cb49fda420fa97"
server_key_iv: "e95e018ba1b5cd4a8b198659bebb4d59"
server_key_version: "encryptionkey@0"
I20260925 09:19:17.545707 25547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "d818fcc6fc044378a8f6a662e3d22962"
format_stamp: "Formatted at 2026-09-25 09:19:17 on dist-test-slave-l6qh"
server_key: "ff7b3df6bf54753892cb49fda420fa97"
server_key_iv: "e95e018ba1b5cd4a8b198659bebb4d59"
server_key_version: "encryptionkey@0"
I20260925 09:19:17.551026 25547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:19:17.554631 25562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:17.556071 25547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:19:17.556255 25547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "d818fcc6fc044378a8f6a662e3d22962"
format_stamp: "Formatted at 2026-09-25 09:19:17 on dist-test-slave-l6qh"
server_key: "ff7b3df6bf54753892cb49fda420fa97"
server_key_iv: "e95e018ba1b5cd4a8b198659bebb4d59"
server_key_version: "encryptionkey@0"
I20260925 09:19:17.556448 25547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:17.562458 25547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:17.566816 25547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:17.567126 25547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:17.586341 25547 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:34725
I20260925 09:19:17.586388 25613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:34725 every 8 connection(s)
I20260925 09:19:17.587749 25547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:19:17.588963 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 25547
I20260925 09:19:17.589166 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:19:17.589712 3149 external_mini_cluster.cc:1428] Setting key d55117dc957e5f12b8e163d78e0ad0bd
I20260925 09:19:17.592427 25614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:17.598479 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:17.591457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52719 (local address 127.3.19.126:34725)
0925 09:19:17.592248 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:17.592262 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:17.592309 (+ 47us) server_negotiation.cc:407] Connection header received
0925 09:19:17.594596 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:17.594623 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:17.595030 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:17.595466 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:17.596458 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:17.596466 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:17.596481 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:19:17.596498 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:17.596587 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:19:17.596628 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:17.596851 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:17.597060 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 09:19:17.597274 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":298,"threads_started":1}
I20260925 09:19:17.602228 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:19:17.605728 25614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962: Bootstrap starting.
I20260925 09:19:17.609270 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:17.606590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35589 (local address 127.3.19.126:34725)
0925 09:19:17.606775 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:17.606781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:17.607252 (+ 471us) server_negotiation.cc:407] Connection header received
0925 09:19:17.607388 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:17.607395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:17.607460 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:17.607578 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:17.608522 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:17.608529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:17.608535 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 09:19:17.608554 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:17.608602 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 09:19:17.608622 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:17.608758 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:17.608891 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 09:19:17.608948 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 09:19:17.609514 25614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:17.610661 25614 log.cc:824] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:17.613656 25614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962: No bootstrap required, opened a new log
I20260925 09:19:17.621042 25614 raft_consensus.cc:348] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d818fcc6fc044378a8f6a662e3d22962" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34725 } }
I20260925 09:19:17.621536 25614 raft_consensus.cc:374] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 09:19:17.621660 25614 raft_consensus.cc:729] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d818fcc6fc044378a8f6a662e3d22962, State: Initialized, Role: FOLLOWER
I20260925 09:19:17.622339 25614 consensus_queue.cc:260] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d818fcc6fc044378a8f6a662e3d22962" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34725 } }
I20260925 09:19:17.622606 25614 raft_consensus.cc:388] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 09:19:17.622763 25614 raft_consensus.cc:482] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 09:19:17.622941 25614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:17.625281 25614 raft_consensus.cc:504] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d818fcc6fc044378a8f6a662e3d22962" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34725 } }
I20260925 09:19:17.625839 25614 leader_election.cc:302] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d818fcc6fc044378a8f6a662e3d22962; no voters:
I20260925 09:19:17.626437 25614 leader_election.cc:288] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 09:19:17.626585 25624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:19:17.627279 25624 raft_consensus.cc:686] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [term 1 LEADER]: Becoming Leader. State: Replica: d818fcc6fc044378a8f6a662e3d22962, State: Running, Role: LEADER
I20260925 09:19:17.627799 25624 consensus_queue.cc:237] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d818fcc6fc044378a8f6a662e3d22962" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34725 } }
I20260925 09:19:17.628564 25614 sys_catalog.cc:564] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:17.630867 25626 sys_catalog.cc:455] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d818fcc6fc044378a8f6a662e3d22962. Latest consensus state: current_term: 1 leader_uuid: "d818fcc6fc044378a8f6a662e3d22962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d818fcc6fc044378a8f6a662e3d22962" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34725 } } }
I20260925 09:19:17.631175 25625 sys_catalog.cc:455] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d818fcc6fc044378a8f6a662e3d22962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d818fcc6fc044378a8f6a662e3d22962" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 34725 } } }
I20260925 09:19:17.631353 25626 sys_catalog.cc:458] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:17.631433 25625 sys_catalog.cc:458] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962 [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:17.634088 25633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:19:17.639178 25633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 09:19:17.649128 25633 catalog_manager.cc:1347] Generated new cluster ID: ad7025ff344d4efbae05bd6019d2013b
I20260925 09:19:17.649374 25633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:19:17.678558 25633 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:19:17.679018 25633 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:19:17.697190 25633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d818fcc6fc044378a8f6a662e3d22962: Generated new TSK 0
I20260925 09:19:17.698143 25633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:19:17.716790 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 25547
W20260925 09:19:17.718053 25609 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52719
W20260925 09:19:17.718848 3149 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 25547: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 09:19:17.719012 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 25547
2026-09-25T09:19:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1970 ms)
[----------] 18 tests from AuthTokenIssuingTest (35606 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:19:17 dist-test-slave-l6qh krb5kdc[25645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:19:17 dist-test-slave-l6qh krb5kdc[25645](info): set up 2 sockets
Sep 25 09:19:17 dist-test-slave-l6qh krb5kdc[25645](info): commencing operation
krb5kdc: starting...
W20260925 09:19:18.834883 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:19:19 dist-test-slave-l6qh krb5kdc[25645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:19:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:19:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:19:19.122925 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:43111
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:43111,127.3.19.125:34067,127.3.19.124:35987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:19:19.456383 25658 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 09:19:19.456754 25658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:19.456878 25658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:19.457026 25658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:19.464401 25658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:19.464491 25658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:19.464572 25658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:19.464644 25658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:19.464732 25658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:19.474740 25658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:43111,127.3.19.125:34067,127.3.19.124:35987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:43111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:19.476334 25658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:19.477874 25658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:19:19.490736 25663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:19.490708 25664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:19.491101 25666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:19.491525 25658 server_base.cc:1034] running on GCE node
I20260925 09:19:19.492257 25658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:19.493662 25658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:19.494879 25658 hybrid_clock.cc:648] HybridClock initialized: now 1790327959494827 us; error 26 us; skew 500 ppm
I20260925 09:19:19.495177 25658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:19 dist-test-slave-l6qh krb5kdc[25645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327959, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:19.498140 25658 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:19:19.499605 25658 webserver.cc:460] Webserver started at http://127.3.19.126:33713/ using document root <none> and password file <none>
I20260925 09:19:19.500228 25658 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:19.500311 25658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:19.500562 25658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:19.503479 25658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "6d8e918ee28448ec8f77ca59ea10461d"
format_stamp: "Formatted at 2026-09-25 09:19:19 on dist-test-slave-l6qh"
server_key: "9d981e0566f82ca69f70fc0a264dc600"
server_key_iv: "57d060db9f92e451f9795d432a1382e6"
server_key_version: "encryptionkey@0"
I20260925 09:19:19.504189 25658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "6d8e918ee28448ec8f77ca59ea10461d"
format_stamp: "Formatted at 2026-09-25 09:19:19 on dist-test-slave-l6qh"
server_key: "9d981e0566f82ca69f70fc0a264dc600"
server_key_iv: "57d060db9f92e451f9795d432a1382e6"
server_key_version: "encryptionkey@0"
I20260925 09:19:19.509053 25658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 09:19:19.512409 25673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:19.513743 25658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:19:19.513870 25658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "6d8e918ee28448ec8f77ca59ea10461d"
format_stamp: "Formatted at 2026-09-25 09:19:19 on dist-test-slave-l6qh"
server_key: "9d981e0566f82ca69f70fc0a264dc600"
server_key_iv: "57d060db9f92e451f9795d432a1382e6"
server_key_version: "encryptionkey@0"
I20260925 09:19:19.514041 25658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:19.537127 25658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:19.541276 25658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:19.541502 25658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:19.561568 25658 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:43111
I20260925 09:19:19.561546 25724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:43111 every 8 connection(s)
I20260925 09:19:19.562983 25658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:19:19.566671 25725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:19.570000 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 25658
I20260925 09:19:19.570232 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:19:19.571015 3149 external_mini_cluster.cc:1428] Setting key b7b2342f4cd2068cb55ad6200c67ec2a
I20260925 09:19:19.575399 25725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:19:19 dist-test-slave-l6qh krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327959, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.19.125@KRBTEST.COM; defaulting to no policy
I20260925 09:19:19.598886 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:19.579330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.3.19.126:43111)
0925 09:19:19.581183 (+ 1853us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:19.581189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:19.581274 (+ 85us) server_negotiation.cc:407] Connection header received
0925 09:19:19.583445 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:19.583454 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:19.583810 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:19.584049 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:19.585153 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:19.586393 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:19.587751 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:19.588295 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:19.588500 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:19.591693 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:19.591700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:19.591710 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:19.591736 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:19.594480 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:19.595163 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:19.595173 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:19.595182 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:19.595321 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:19.595716 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:19.595723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:19.595727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:19.596197 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:19.596405 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:19.596990 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 09:19:19.597420 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":1666,"thread_start_us":1396,"threads_started":1}
I20260925 09:19:19.598883 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:19.579330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43111 (local address 127.0.0.1:45850)
0925 09:19:19.580240 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:19.580298 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:19.583062 (+ 2764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:19.584057 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:19.584061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:19.584598 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:19.584917 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:19.584932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:19.586559 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:19.586569 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:19.587611 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:19.587623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:19.588535 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:19.588542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:19.588723 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:19.589393 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:19.589420 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:19.591472 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:19.594633 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:19.594646 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:19.594657 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:19.595027 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:19.595433 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:19.595440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:19.595444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:19.595606 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:19.596333 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:19.596341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:19.596613 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 09:19:19.596877 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 09:19:19.597530 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":167,"thread_start_us":377,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.125@KRBTEST.COM" created.
I20260925 09:19:19.602062 25725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } has no permanent_uuid. Determining permanent_uuid...
I20260925 09:19:19.603123 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:19.602720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:34067 (local address 127.0.0.1:41988)
0925 09:19:19.602892 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:19.603006 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 09:19:19.603385 25676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
I20260925 09:19:19.604866 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:19.604540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:34067 (local address 127.0.0.1:41998)
0925 09:19:19.604687 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:19.604755 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 09:19:19.605082 25725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:34067: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.125.keytab.
Entry for principal kudu/127.3.19.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.125.keytab.
I20260925 09:19:19.632166 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.125:34067
--webserver_interface=127.3.19.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.125.keytab
--principal=kudu/127.3.19.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:43111,127.3.19.125:34067,127.3.19.124:35987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:19:19.654480 25725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } attempt: 1
I20260925 09:19:19.656045 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:19.655358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:34067 (local address 127.0.0.1:42014)
0925 09:19:19.655710 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:19.655835 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260925 09:19:19.657272 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:19.656864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:34067 (local address 127.0.0.1:42028)
0925 09:19:19.657011 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:19.657084 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 09:19:19.657608 25725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:34067: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
I20260925 09:19:19.725901 25725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } attempt: 2
I20260925 09:19:19.727146 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:19.726627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:34067 (local address 127.0.0.1:42040)
0925 09:19:19.726894 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:19.726997 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 09:19:19.728288 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:19.727911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:34067 (local address 127.0.0.1:42050)
0925 09:19:19.728086 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:19.728170 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260925 09:19:19.728569 25725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:34067: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
I20260925 09:19:19.819816 25725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } attempt: 3
I20260925 09:19:19.821049 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:19.820598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:34067 (local address 127.0.0.1:42056)
0925 09:19:19.820809 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:19.820902 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 09:19:19.822124 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:19.821813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:34067 (local address 127.0.0.1:42070)
0925 09:19:19.821937 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:19.822001 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 09:19:19.822373 25725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:34067: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
I20260925 09:19:19.965618 25725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } attempt: 4
I20260925 09:19:19.967029 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:19.966418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:34067 (local address 127.0.0.1:42086)
0925 09:19:19.966798 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:19.966885 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260925 09:19:19.968089 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:19.967715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:34067 (local address 127.0.0.1:42102)
0925 09:19:19.967883 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:19.967967 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 09:19:19.968266 25725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:34067: Network error: Client connection negotiation failed: client connection to 127.3.19.125:34067: connect: Connection refused (error 111)
W20260925 09:19:19.973860 25732 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 09:19:19.974188 25732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:19.974316 25732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:19.974453 25732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:19.981578 25732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:19.981700 25732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:19.981817 25732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:19.981905 25732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:19.982014 25732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:19.991658 25732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:43111,127.3.19.125:34067,127.3.19.124:35987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.125.keytab
--principal=kudu/127.3.19.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.125:34067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.19.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:19.993048 25732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:19.994367 25732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:19:20.006377 25740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:20.006501 25737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:20.006451 25738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:20.010934 25732 server_base.cc:1034] running on GCE node
I20260925 09:19:20.011821 25732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:20.013409 25732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:20.014685 25732 hybrid_clock.cc:648] HybridClock initialized: now 1790327960014625 us; error 41 us; skew 500 ppm
I20260925 09:19:20.015034 25732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:20 dist-test-slave-l6qh krb5kdc[25645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:20.018422 25732 init.cc:377] Logged in from keytab as kudu/127.3.19.125@KRBTEST.COM (short username kudu)
I20260925 09:19:20.019964 25732 webserver.cc:460] Webserver started at http://127.3.19.125:40585/ using document root <none> and password file <none>
I20260925 09:19:20.020629 25732 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:20.020716 25732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:20.020969 25732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:20.023900 25732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/data/instance:
uuid: "28eea4b2a4184275b7119ecd438ca782"
format_stamp: "Formatted at 2026-09-25 09:19:20 on dist-test-slave-l6qh"
server_key: "35ed70d2292a4acd789e2cc65ba6425d"
server_key_iv: "4940f5e702315e7d8ca59009b281a11f"
server_key_version: "encryptionkey@0"
I20260925 09:19:20.024621 25732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/wal/instance:
uuid: "28eea4b2a4184275b7119ecd438ca782"
format_stamp: "Formatted at 2026-09-25 09:19:20 on dist-test-slave-l6qh"
server_key: "35ed70d2292a4acd789e2cc65ba6425d"
server_key_iv: "4940f5e702315e7d8ca59009b281a11f"
server_key_version: "encryptionkey@0"
I20260925 09:19:20.029806 25732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 09:19:20.033071 25747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:20.034426 25732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 09:19:20.034559 25732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/wal
uuid: "28eea4b2a4184275b7119ecd438ca782"
format_stamp: "Formatted at 2026-09-25 09:19:20 on dist-test-slave-l6qh"
server_key: "35ed70d2292a4acd789e2cc65ba6425d"
server_key_iv: "4940f5e702315e7d8ca59009b281a11f"
server_key_version: "encryptionkey@0"
I20260925 09:19:20.034736 25732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:20.059286 25732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:20.063868 25732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:20.064113 25732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:20.082634 25732 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.125:34067
I20260925 09:19:20.082655 25798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.125:34067 every 8 connection(s)
I20260925 09:19:20.083972 25732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/data/info.pb
I20260925 09:19:20.087502 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 25732
I20260925 09:19:20.087673 25799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:20.087755 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-1/wal/instance
I20260925 09:19:20.088173 3149 external_mini_cluster.cc:1428] Setting key 1fc75af8030060e752b406ec718c6877
I20260925 09:19:20.094551 25799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:19:20 dist-test-slave-l6qh krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.125@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.19.124@KRBTEST.COM; defaulting to no policy
I20260925 09:19:20.115186 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.097132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.3.19.126:43111)
0925 09:19:20.097415 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.097425 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.098698 (+ 1273us) server_negotiation.cc:407] Connection header received
0925 09:19:20.100808 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.100814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.100911 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.101033 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.102653 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.103389 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.104953 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.105447 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.105590 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.109064 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.109069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.109072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.109106 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.111981 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.112731 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.112739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.112747 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.112862 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.113303 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.113310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.113313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.113597 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.113762 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.114613 (+ 851us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.114835 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.124@KRBTEST.COM" created.
I20260925 09:19:20.116454 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.097652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43111 (local address 127.0.0.1:45864)
0925 09:19:20.098485 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.098552 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:20.100523 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:20.101352 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:20.101361 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:20.101875 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:20.102422 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.102440 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.103570 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.103574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.104782 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.104794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.105703 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.105711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.105939 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.106644 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:20.106671 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:20.108817 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:20.112138 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.112147 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.112163 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.112595 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.112976 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.112983 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.112987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.113165 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.113732 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:20.113740 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:20.114236 (+ 496us) client_negotiation.cc:769] Sending connection context
0925 09:19:20.114502 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 09:19:20.115099 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":316,"threads_started":1}
I20260925 09:19:20.118202 25799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:19:20 dist-test-slave-l6qh krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.125@KRBTEST.COM for kudu/127.3.19.125@KRBTEST.COM
I20260925 09:19:20.134714 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.119064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:34067 (local address 127.0.0.1:42106)
0925 09:19:20.119316 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.119337 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:20.119441 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:20.120370 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:20.120383 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:20.120874 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:20.121084 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.121099 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.122580 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.122589 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.124087 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.124105 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.124936 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.124948 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.125172 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.125805 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:20.125830 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:20.127795 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:20.132238 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.132242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.132244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.132671 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.133089 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.133093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.133095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.133294 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.134037 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:20.134044 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:20.134155 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 09:19:20.134263 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 09:19:20.134450 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 09:19:20.135247 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.119308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.3.19.125:34067)
0925 09:19:20.119963 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.119969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.119995 (+ 26us) server_negotiation.cc:407] Connection header received
0925 09:19:20.120039 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.120045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.120212 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.120323 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.121310 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.122413 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.124235 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.124779 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.124877 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.128015 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.128031 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.128052 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.128093 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.132065 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.132807 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.132819 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.132825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.132974 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.133411 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.133419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.133425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.133911 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.134132 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.134444 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.134740 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":362,"threads_started":1}
I20260925 09:19:20.137713 25799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } has no permanent_uuid. Determining permanent_uuid...
I20260925 09:19:20.138904 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.138429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:48974)
0925 09:19:20.138600 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.138770 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 09:19:20.139133 25749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
I20260925 09:19:20.140770 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.140465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:48984)
0925 09:19:20.140599 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.140672 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 09:19:20.141005 25799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:35987: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.124.keytab.
Entry for principal kudu/127.3.19.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.124.keytab.
I20260925 09:19:20.153635 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.124:35987
--webserver_interface=127.3.19.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.124.keytab
--principal=kudu/127.3.19.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:43111,127.3.19.125:34067,127.3.19.124:35987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:19:20.190313 25799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } attempt: 1
I20260925 09:19:20.191531 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.191048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:48994)
0925 09:19:20.191244 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.191352 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 09:19:20.192508 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.192150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:48998)
0925 09:19:20.192325 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.192397 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 09:19:20.192714 25799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:35987: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
I20260925 09:19:20.260915 25799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } attempt: 2
I20260925 09:19:20.262204 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.261703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49008)
0925 09:19:20.261937 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.262031 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260925 09:19:20.263226 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.262895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49014)
0925 09:19:20.263039 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.263116 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 09:19:20.263427 25799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:35987: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
I20260925 09:19:20.267513 25725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } attempt: 5
Sep 25 09:19:20 dist-test-slave-l6qh krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327959, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for kudu/127.3.19.125@KRBTEST.COM
I20260925 09:19:20.280500 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.268137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:34067 (local address 127.0.0.1:42112)
0925 09:19:20.268305 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.268323 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:20.268460 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:20.268839 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:20.268855 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:20.269327 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:20.269510 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.269519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.270499 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.270505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.271370 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.271381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.272110 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.272116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.272327 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.272949 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:20.272966 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:20.275161 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:20.278078 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.278085 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.278089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.278453 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.278838 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.278843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.278847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.278990 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.279552 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:20.279557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:20.279702 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 09:19:20.279872 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 09:19:20.280144 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 09:19:20.280503 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.268348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42112 (local address 127.3.19.125:34067)
0925 09:19:20.268519 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.268528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.268558 (+ 30us) server_negotiation.cc:407] Connection header received
0925 09:19:20.268618 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.268621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.268694 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.268802 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.269660 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.270363 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.271519 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.271948 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.272082 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.275347 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.275351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.275355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.275384 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.277938 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.278597 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.278604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.278607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.278726 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.279104 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.279111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.279114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.279421 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.279579 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.279968 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.280178 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 09:19:20.281740 25725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } has no permanent_uuid. Determining permanent_uuid...
I20260925 09:19:20.282802 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.282442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49028)
0925 09:19:20.282597 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.282676 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 09:19:20.283767 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.283459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49044)
0925 09:19:20.283602 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.283678 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 09:19:20.283999 25725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:35987: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
I20260925 09:19:20.335245 25725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } attempt: 1
I20260925 09:19:20.336467 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.336016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49056)
0925 09:19:20.336209 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.336300 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 09:19:20.337493 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.337111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49066)
0925 09:19:20.337327 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.337395 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 09:19:20.337693 25725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:35987: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
I20260925 09:19:20.354635 25799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } attempt: 3
I20260925 09:19:20.355763 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.355228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49078)
0925 09:19:20.355457 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.355606 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 09:19:20.356834 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.356454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49092)
0925 09:19:20.356666 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.356742 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260925 09:19:20.357019 25799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:35987: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
I20260925 09:19:20.405887 25725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } attempt: 2
I20260925 09:19:20.407039 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.406503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49094)
0925 09:19:20.406740 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.406862 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 09:19:20.407943 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.407634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49102)
0925 09:19:20.407770 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.407840 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 09:19:20.408152 25725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:35987: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
W20260925 09:19:20.489660 25806 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 09:19:20.490025 25806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:20.490180 25806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:20.490314 25806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:20.497432 25806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:20.497521 25806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:20.497668 25806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:20.497761 25806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:20.497882 25806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:20.500224 25799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } attempt: 4
I20260925 09:19:20.501379 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.500853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49116)
0925 09:19:20.501082 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.501164 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 09:19:20.502313 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.502002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49124)
0925 09:19:20.502153 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.502220 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 09:19:20.502498 25799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:35987: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
I20260925 09:19:20.508224 25806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:43111,127.3.19.125:34067,127.3.19.124:35987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.124.keytab
--principal=kudu/127.3.19.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.124:35987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.19.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:20.509631 25806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:20.510929 25806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 09:19:20.514341 25725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } attempt: 3
I20260925 09:19:20.515268 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.514874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49138)
0925 09:19:20.515048 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.515132 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 09:19:20.516304 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.515860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49150)
0925 09:19:20.516083 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.516146 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260925 09:19:20.516542 25725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:35987: Network error: Client connection negotiation failed: client connection to 127.3.19.124:35987: connect: Connection refused (error 111)
W20260925 09:19:20.523804 25814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:20.523835 25811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:20.523804 25812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:20.524447 25806 server_base.cc:1034] running on GCE node
I20260925 09:19:20.525386 25806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:20.527070 25806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:20.528298 25806 hybrid_clock.cc:648] HybridClock initialized: now 1790327960528232 us; error 36 us; skew 500 ppm
I20260925 09:19:20.528699 25806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:20 dist-test-slave-l6qh krb5kdc[25645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:20.531980 25806 init.cc:377] Logged in from keytab as kudu/127.3.19.124@KRBTEST.COM (short username kudu)
I20260925 09:19:20.533448 25806 webserver.cc:460] Webserver started at http://127.3.19.124:38125/ using document root <none> and password file <none>
I20260925 09:19:20.534085 25806 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:20.534170 25806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:20.534431 25806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:20.537374 25806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/data/instance:
uuid: "c244f6fc639c42e69c701113e879d943"
format_stamp: "Formatted at 2026-09-25 09:19:20 on dist-test-slave-l6qh"
server_key: "bfe298f6e3a1c40b4ffdb8dd712cbea6"
server_key_iv: "2a27fd34c296216e9e3cb299e754cdc8"
server_key_version: "encryptionkey@0"
I20260925 09:19:20.538260 25806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/wal/instance:
uuid: "c244f6fc639c42e69c701113e879d943"
format_stamp: "Formatted at 2026-09-25 09:19:20 on dist-test-slave-l6qh"
server_key: "bfe298f6e3a1c40b4ffdb8dd712cbea6"
server_key_iv: "2a27fd34c296216e9e3cb299e754cdc8"
server_key_version: "encryptionkey@0"
I20260925 09:19:20.543428 25806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:19:20.546967 25821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:20.548233 25806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:19:20.548426 25806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/wal
uuid: "c244f6fc639c42e69c701113e879d943"
format_stamp: "Formatted at 2026-09-25 09:19:20 on dist-test-slave-l6qh"
server_key: "bfe298f6e3a1c40b4ffdb8dd712cbea6"
server_key_iv: "2a27fd34c296216e9e3cb299e754cdc8"
server_key_version: "encryptionkey@0"
I20260925 09:19:20.548655 25806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:20.573211 25806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:20.577821 25806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:20.578102 25806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:20.597055 25806 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.124:35987
I20260925 09:19:20.597074 25872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.124:35987 every 8 connection(s)
I20260925 09:19:20.598372 25806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/data/info.pb
I20260925 09:19:20.600399 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 25806
I20260925 09:19:20.600602 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790327600440376-3149-0/minicluster-data/master-2/wal/instance
I20260925 09:19:20.601047 3149 external_mini_cluster.cc:1428] Setting key 95c8b2dcc98bee2165d792f75b06948c
I20260925 09:19:20.602451 25873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:20.611693 25873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:19:20 dist-test-slave-l6qh krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.124@KRBTEST.COM
Sep 25 09:19:20 dist-test-slave-l6qh krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.124@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:19:20.624011 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.602887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.3.19.124:35987)
0925 09:19:20.603721 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.603736 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.603806 (+ 70us) server_negotiation.cc:407] Connection header received
0925 09:19:20.605558 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.605578 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.605935 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.606263 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.608073 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.609440 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.611224 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.611900 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.612023 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.616584 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.616592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.616601 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.616637 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.619839 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.620624 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.620631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.620637 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.620839 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.621271 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.621276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.621281 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.621832 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.622135 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.622475 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.622881 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"spinlock_wait_cycles":12160,"thread_start_us":400,"threads_started":1}
I20260925 09:19:20.628803 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.614261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:43111 (local address 127.0.0.1:45872)
0925 09:19:20.615062 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.615100 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:20.615249 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:20.615676 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:20.615686 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:20.616238 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:20.616506 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.616514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.617773 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.617778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.619023 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.619038 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.619857 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.619863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.620031 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.620594 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:20.620643 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:20.623033 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:20.626093 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.626103 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.626109 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.626572 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.627053 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.627058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.627061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.627205 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.627743 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:20.627751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:20.627925 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 09:19:20.628102 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 09:19:20.628386 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":432,"threads_started":1}
I20260925 09:19:20.628803 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.614322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.3.19.126:43111)
0925 09:19:20.614548 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.614555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.615211 (+ 656us) server_negotiation.cc:407] Connection header received
0925 09:19:20.615379 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.615385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.615461 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.615583 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.616664 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.617620 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.619214 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.619661 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.619798 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.623210 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.623214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.623218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.623248 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.625969 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.626718 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.626724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.626727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.626926 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.627329 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.627336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.627339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.627612 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.627786 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.628212 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.628413 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 09:19:20.630527 25873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:19:20 dist-test-slave-l6qh krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
Sep 25 09:19:20 dist-test-slave-l6qh krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.124@KRBTEST.COM for kudu/127.3.19.125@KRBTEST.COM
I20260925 09:19:20.644731 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.626853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.3.19.126:43111)
0925 09:19:20.629368 (+ 2515us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.629373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.629388 (+ 15us) server_negotiation.cc:407] Connection header received
0925 09:19:20.629431 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.629438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.629500 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.629612 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.631651 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.633117 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.634469 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.634899 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.635041 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.639352 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.639357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.639359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.639387 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.642057 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.642811 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.642815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.642817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.642928 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.643321 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.643324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.643326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.643592 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.643876 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.644200 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.644401 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2413,"thread_start_us":10559,"threads_started":1}
I20260925 09:19:20.646188 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.631355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:34067 (local address 127.0.0.1:42116)
0925 09:19:20.631549 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.631567 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:20.631661 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:20.631972 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:20.631977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:20.632331 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:20.632458 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.632466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.633576 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.633585 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.635481 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.635507 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.637017 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.637026 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.637331 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.637942 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:20.637958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:20.640155 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:20.643685 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.643690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.643694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.644037 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.644438 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.644443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.644447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.644659 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.645292 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:20.645297 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:20.645449 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 09:19:20.645596 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 09:19:20.645854 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 09:19:20.646272 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.631331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.3.19.125:34067)
0925 09:19:20.631526 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.631531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.631614 (+ 83us) server_negotiation.cc:407] Connection header received
0925 09:19:20.631789 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.631792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.631853 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.631957 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.632632 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.633412 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.635703 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.636598 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.636715 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.640348 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.640353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.640356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.640387 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.643554 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.644160 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.644165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.644169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.644316 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.644754 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.644759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.644763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.645134 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.645327 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.645683 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.645887 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 09:19:20.647454 25873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:19:20 dist-test-slave-l6qh krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.125@KRBTEST.COM
Sep 25 09:19:20 dist-test-slave-l6qh krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.124@KRBTEST.COM for kudu/127.3.19.124@KRBTEST.COM
I20260925 09:19:20.661734 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.646559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.3.19.125:34067)
0925 09:19:20.646733 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.646740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.646759 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:19:20.646806 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.646810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.646875 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.647007 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.648394 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.649110 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.650336 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.650755 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.650895 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.655397 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.655402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.655405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.655458 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.658844 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.659654 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.659659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.659663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.659835 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.660257 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.660262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.660265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.660592 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.660855 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.661151 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.661401 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 09:19:20.661767 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.648217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49176)
0925 09:19:20.648443 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.648455 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:20.648591 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:20.648893 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:20.648897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:20.649303 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:20.649438 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.649446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.650368 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.650375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.651212 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.651223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.652244 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.652248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.652554 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.653089 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:20.653106 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:20.655379 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:20.658972 (+ 3593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.658977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.658981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.659529 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.659965 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.659970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.659973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.660135 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.660722 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:20.660727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:20.660880 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 09:19:20.661044 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 09:19:20.661379 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260925 09:19:20.662180 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.648150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.3.19.124:35987)
0925 09:19:20.648325 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.648331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.648532 (+ 201us) server_negotiation.cc:407] Connection header received
0925 09:19:20.648705 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.648709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.648787 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.648879 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.649570 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.650247 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.651376 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.652077 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.652212 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.655538 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.655554 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.655559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.655604 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.658811 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.659654 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.659659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.659663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.659839 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.660257 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.660262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.660265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.660595 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.660732 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.661177 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.661598 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":58}
I20260925 09:19:20.666308 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:19:20.667390 25873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943: Bootstrap starting.
I20260925 09:19:20.671392 25873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:20.672713 25873 log.cc:824] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:20.668895 25700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6d8e918ee28448ec8f77ca59ea10461d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45884
I20260925 09:19:20.675774 25873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943: No bootstrap required, opened a new log
I20260925 09:19:20.686436 25873 raft_consensus.cc:348] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } }
I20260925 09:19:20.686812 25873 raft_consensus.cc:729] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c244f6fc639c42e69c701113e879d943, State: Initialized, Role: FOLLOWER
I20260925 09:19:20.687299 25873 consensus_queue.cc:260] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } }
I20260925 09:19:20.688421 25880 sys_catalog.cc:455] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } } }
I20260925 09:19:20.688757 25880 sys_catalog.cc:458] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:19:20.689464 25873 sys_catalog.cc:564] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:20.693770 25725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } attempt: 4
W20260925 09:19:20.698902 25891 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 09:19:20.699110 25891 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 09:19:20 dist-test-slave-l6qh krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327959, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for kudu/127.3.19.124@KRBTEST.COM
I20260925 09:19:20.697177 25806 master_runner.cc:186] Error getting master registration for 127.3.19.126:43111: OK, instance_id { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" instance_seqno: 1790327959555974 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 09:19:20.708216 25806 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 09:19:20.712126 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.694698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.3.19.124:35987)
0925 09:19:20.694856 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.694864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.694885 (+ 21us) server_negotiation.cc:407] Connection header received
0925 09:19:20.694939 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.694946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.695029 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.695133 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.696214 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.697482 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.699403 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.700171 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.700294 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.704888 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.704895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.704900 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.704942 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.708857 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.709835 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.709841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.709844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.709989 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.710455 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.710467 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.710471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.710873 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.711036 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.711527 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.711746 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 09:19:20.712211 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.694360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49192)
0925 09:19:20.694558 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.694579 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:20.694701 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:20.695315 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:20.695321 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:20.695852 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:20.696065 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.696074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.697658 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.697666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.699230 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.699248 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.700538 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.700544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.700790 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.701525 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:20.701549 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:20.704576 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:20.709019 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.709024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.709030 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.709686 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.710118 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.710124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.710128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.710324 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.711007 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:20.711013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:20.711229 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 09:19:20.711401 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 09:19:20.711728 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 09:19:20.718410 25725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d: Bootstrap starting.
I20260925 09:19:20.722642 25725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:20.724025 25725 log.cc:824] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:20.727734 25725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d: No bootstrap required, opened a new log
I20260925 09:19:20.735534 25725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } }
I20260925 09:19:20.735996 25725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8e918ee28448ec8f77ca59ea10461d, State: Initialized, Role: FOLLOWER
I20260925 09:19:20.736547 25725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } }
I20260925 09:19:20.737586 25894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } } }
I20260925 09:19:20.738063 25894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:19:20.738950 25725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:20.739037 25700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 09:19:20.739254 25700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:20.742022 25700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } }
I20260925 09:19:20.743294 25700 leader_election.cc:288] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [CANDIDATE]: Term 1 election: Requested vote from peers 28eea4b2a4184275b7119ecd438ca782 (127.3.19.125:34067), c244f6fc639c42e69c701113e879d943 (127.3.19.124:35987)
I20260925 09:19:20.744388 25848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d8e918ee28448ec8f77ca59ea10461d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c244f6fc639c42e69c701113e879d943"
I20260925 09:19:20.744907 25848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:20.749082 25848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8e918ee28448ec8f77ca59ea10461d in term 1.
I20260925 09:19:20.749848 25675 leader_election.cc:302] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d8e918ee28448ec8f77ca59ea10461d, c244f6fc639c42e69c701113e879d943; no voters:
I20260925 09:19:20.750368 25894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:19:20.757889 25894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [term 1 LEADER]: Becoming Leader. State: Replica: 6d8e918ee28448ec8f77ca59ea10461d, State: Running, Role: LEADER
I20260925 09:19:20.758580 25894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } }
I20260925 09:19:20.765939 25899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d8e918ee28448ec8f77ca59ea10461d. Latest consensus state: current_term: 1 leader_uuid: "6d8e918ee28448ec8f77ca59ea10461d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } } }
I20260925 09:19:20.766444 25899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:20.773842 25903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:19:20.788475 25903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 09:19:20.794277 25679 tablet.cc:2369] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:19:20.801719 25799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } attempt: 5
I20260925 09:19:20.822211 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.778008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.3.19.126:43111)
0925 09:19:20.778237 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.778246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.781796 (+ 3550us) server_negotiation.cc:407] Connection header received
0925 09:19:20.781994 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.782002 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.782105 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.782242 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.791896 (+ 9654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.793089 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.796236 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.796978 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.797098 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.800667 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.800674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.800681 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.800723 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.807245 (+ 6522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.818832 (+ 11587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.818843 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.818848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.819053 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.819907 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.819913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.819916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.820270 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.820549 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.821547 (+ 998us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.821817 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 09:19:20.826362 25848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943 [term 1 FOLLOWER]: Refusing update from remote peer 6d8e918ee28448ec8f77ca59ea10461d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:19:20.827448 25899 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 09:19:20.830217 25827 tablet.cc:2369] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:19:20.837803 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.782157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.3.19.124:35987)
0925 09:19:20.782328 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.782333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.783975 (+ 1642us) server_negotiation.cc:407] Connection header received
0925 09:19:20.784574 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.784581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.784710 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.784799 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.792400 (+ 7601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.793856 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.804642 (+ 10786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.805656 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.805782 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.821521 (+ 15739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.821532 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.821540 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.821595 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.827694 (+ 6099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.830825 (+ 3131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.830865 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.830871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.831235 (+ 364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.832806 (+ 1571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.832811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.832815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.833605 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.834092 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.836217 (+ 2125us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.836782 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 09:19:20 dist-test-slave-l6qh krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.125@KRBTEST.COM for kudu/127.3.19.124@KRBTEST.COM
I20260925 09:19:20.851364 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.783127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42136 (local address 127.3.19.125:34067)
0925 09:19:20.783297 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.783303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.788966 (+ 5663us) server_negotiation.cc:407] Connection header received
0925 09:19:20.789851 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.789857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.789950 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.790030 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.801063 (+ 11033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.802544 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.812860 (+ 10316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.813778 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.813902 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.839425 (+ 25523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.839434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.839442 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.839498 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.844154 (+ 4656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.846963 (+ 2809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.846972 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.846977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.847302 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.848163 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.848170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.848175 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.848642 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.848927 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.850143 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.850602 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 09:19:20.858539 25880 sys_catalog.cc:455] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d8e918ee28448ec8f77ca59ea10461d. Latest consensus state: current_term: 1 leader_uuid: "6d8e918ee28448ec8f77ca59ea10461d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } } }
I20260925 09:19:20.858865 25880 sys_catalog.cc:458] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:19:20.865924 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.802252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:35987 (local address 127.0.0.1:49208)
0925 09:19:20.802604 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:20.802624 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:20.802760 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:20.827367 (+ 24607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:20.827372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:20.827843 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:20.828068 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.828084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.838801 (+ 10717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.838807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.840392 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:20.840407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.845364 (+ 4957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.845370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:20.845596 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.846228 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:20.846253 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:20.849014 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:20.861120 (+ 12106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.861125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.861129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.861711 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.862169 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:20.862175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:20.862181 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:20.862363 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:20.863066 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:20.863074 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:20.863250 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 09:19:20.863389 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 09:19:20.864971 (+ 1582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260925 09:19:20.866942 25880 sys_catalog.cc:455] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d8e918ee28448ec8f77ca59ea10461d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } } }
I20260925 09:19:20.867187 25880 sys_catalog.cc:458] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:19:20.868281 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:20.802946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.3.19.124:35987)
0925 09:19:20.824877 (+ 21931us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:20.824885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:20.824910 (+ 25us) server_negotiation.cc:407] Connection header received
0925 09:19:20.824963 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:20.824968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:20.825043 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:20.834413 (+ 9370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:20.834679 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.835908 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.840639 (+ 4731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:20.841327 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:20.849207 (+ 7880us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:20.849441 (+ 234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:20.849447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:20.849452 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:20.849504 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:20.860966 (+ 11462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.861852 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.861858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.861861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.862035 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:20.862548 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:20.862553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:20.862557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:20.862931 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:20.867486 (+ 4555us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:20.867617 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 09:19:20.867834 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21787,"thread_start_us":22185,"threads_started":1}
I20260925 09:19:20.868698 25915 mvcc.cc:205] Tried to move back new op lower bound from 7333183327505817600 to 7333183327273717760. Current Snapshot: MvccSnapshot[applied={T|T < 7333183327505817600 or (T in {7333183327505817600})}]
I20260925 09:19:20.883893 25799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782: Bootstrap starting.
I20260925 09:19:20.904085 25799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:20.911908 25799 log.cc:824] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:20.860644 25658 master_runner.cc:186] Error getting master registration for 127.3.19.125:34067: OK, instance_id { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" instance_seqno: 1790327960077468 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 09:19:20.918635 25658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 09:19:20.919934 25899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d8e918ee28448ec8f77ca59ea10461d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } } }
I20260925 09:19:20.920224 25899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:20.921978 25903 catalog_manager.cc:1347] Generated new cluster ID: c80afabd6d534d8898a409e1310b122c
I20260925 09:19:20.922072 25903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:19:20.927735 25799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782: No bootstrap required, opened a new log
I20260925 09:19:20.940033 25799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } }
I20260925 09:19:20.941710 25799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eea4b2a4184275b7119ecd438ca782, State: Initialized, Role: FOLLOWER
I20260925 09:19:20.942366 25799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } }
I20260925 09:19:20.948608 25903 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:19:20.952471 25924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } } }
I20260925 09:19:20.952876 25924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:19:20.954764 25799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:20.954845 25774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d8e918ee28448ec8f77ca59ea10461d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "28eea4b2a4184275b7119ecd438ca782"
I20260925 09:19:20.955273 25774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:20.960343 25774 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8e918ee28448ec8f77ca59ea10461d in term 1.
I20260925 09:19:20.961508 25773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782 [term 1 FOLLOWER]: Refusing update from remote peer 6d8e918ee28448ec8f77ca59ea10461d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:19:20.963718 25894 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:19:20.976094 25903 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:19:21.022799 25903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d: Generated new TSK 0
I20260925 09:19:21.026849 25903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:19:21.034750 25924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d8e918ee28448ec8f77ca59ea10461d. Latest consensus state: current_term: 1 leader_uuid: "6d8e918ee28448ec8f77ca59ea10461d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } } }
I20260925 09:19:21.035082 25924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:19:21.035159 25894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d8e918ee28448ec8f77ca59ea10461d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } } }
I20260925 09:19:21.035531 25894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8e918ee28448ec8f77ca59ea10461d [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:21.041812 25930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d8e918ee28448ec8f77ca59ea10461d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8e918ee28448ec8f77ca59ea10461d" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 43111 } } peers { permanent_uuid: "28eea4b2a4184275b7119ecd438ca782" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 34067 } } peers { permanent_uuid: "c244f6fc639c42e69c701113e879d943" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 35987 } } }
I20260925 09:19:21.042411 25930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:19:21.073338 25941 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260925 09:19:21.150012 25728 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:21.141565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.3.19.126:43111)
0925 09:19:21.141845 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:21.141855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:21.148143 (+ 6288us) server_negotiation.cc:407] Connection header received
0925 09:19:21.148428 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:21.148435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:21.148513 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:21.148582 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:21.149778 (+ 1196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45892: BlockingRecv error: recv got EOF from 127.0.0.1:45892 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 09:19:21.152880 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:21.147997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.3.19.125:34067)
0925 09:19:21.148190 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:21.148197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:21.151176 (+ 2979us) server_negotiation.cc:407] Connection header received
0925 09:19:21.151410 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:21.151416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:21.151509 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:21.151576 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:21.152676 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42148: BlockingRecv error: recv got EOF from 127.0.0.1:42148 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 09:19:21.154909 25912 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:21.141569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.3.19.124:35987)
0925 09:19:21.142017 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:21.142026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:21.150807 (+ 8781us) server_negotiation.cc:407] Connection header received
0925 09:19:21.153437 (+ 2630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:21.153442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:21.153541 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:21.153615 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:21.154685 (+ 1070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49222: BlockingRecv error: recv got EOF from 127.0.0.1:49222 (error 108)
Metrics: {"server-negotiator.queue_time_us":315}
W20260925 09:19:21.179548 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:21.167148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.3.19.125:34067)
0925 09:19:21.167340 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:21.167348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:21.177825 (+ 10477us) server_negotiation.cc:407] Connection header received
0925 09:19:21.178022 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:21.178028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:21.178160 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:21.178269 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:21.179289 (+ 1020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42152: BlockingRecv error: recv got EOF from 127.0.0.1:42152 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 09:19:21.220082 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:21.214777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.3.19.125:34067)
0925 09:19:21.214986 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:21.214994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:21.218015 (+ 3021us) server_negotiation.cc:407] Connection header received
0925 09:19:21.218486 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:21.218493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:21.218599 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:21.218685 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:21.219820 (+ 1135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42162: BlockingRecv error: recv got EOF from 127.0.0.1:42162 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 09:19:21.267733 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:21.251447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.3.19.125:34067)
0925 09:19:21.251683 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:21.251692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:21.265874 (+ 14182us) server_negotiation.cc:407] Connection header received
0925 09:19:21.266134 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:21.266142 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:21.266248 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:21.266335 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:21.267475 (+ 1140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42172: BlockingRecv error: recv got EOF from 127.0.0.1:42172 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 09:19:21.278285 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:21.276053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.3.19.125:34067)
0925 09:19:21.276278 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:21.276285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:21.276782 (+ 497us) server_negotiation.cc:407] Connection header received
0925 09:19:21.276977 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:21.276995 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:21.277088 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:21.277204 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:21.278061 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42180: BlockingRecv error: recv got EOF from 127.0.0.1:42180 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260925 09:19:21.291887 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:21.289672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.3.19.125:34067)
0925 09:19:21.289849 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:21.289854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:21.290298 (+ 444us) server_negotiation.cc:407] Connection header received
0925 09:19:21.290585 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:21.290591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:21.290692 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:21.290810 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:21.291658 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42186: BlockingRecv error: recv got EOF from 127.0.0.1:42186 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 09:19:21.313577 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:21.311322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.3.19.125:34067)
0925 09:19:21.311504 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:21.311511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:21.312016 (+ 505us) server_negotiation.cc:407] Connection header received
0925 09:19:21.312232 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:21.312238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:21.312326 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:21.312435 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:21.313356 (+ 921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42192: BlockingRecv error: recv got EOF from 127.0.0.1:42192 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 09:19:21.351366 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:21.349210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.3.19.125:34067)
0925 09:19:21.349419 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:21.349425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:21.349940 (+ 515us) server_negotiation.cc:407] Connection header received
0925 09:19:21.350123 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:21.350129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:21.350237 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:21.350352 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:21.351114 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42194: BlockingRecv error: recv got EOF from 127.0.0.1:42194 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260925 09:19:21.420684 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:21.418902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.3.19.125:34067)
0925 09:19:21.419095 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:21.419100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:21.419443 (+ 343us) server_negotiation.cc:407] Connection header received
0925 09:19:21.419610 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:21.419614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:21.419686 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:21.419781 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:21.420439 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42196: BlockingRecv error: recv got EOF from 127.0.0.1:42196 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 09:19:21.554914 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:21.552845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.3.19.125:34067)
0925 09:19:21.553099 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:21.553107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:21.553538 (+ 431us) server_negotiation.cc:407] Connection header received
0925 09:19:21.553759 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:21.553766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:21.553845 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:21.553959 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:21.554679 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42210: BlockingRecv error: recv got EOF from 127.0.0.1:42210 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 09:19:21.701153 25891 catalog_manager.cc:1259] Loaded cluster ID: c80afabd6d534d8898a409e1310b122c
I20260925 09:19:21.701274 25891 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943: loading cluster ID for follower catalog manager: success
I20260925 09:19:21.703732 25891 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943: acquiring CA information for follower catalog manager: success
I20260925 09:19:21.705464 25891 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c244f6fc639c42e69c701113e879d943: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 09:19:21.822331 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:21.820139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42222 (local address 127.3.19.125:34067)
0925 09:19:21.820341 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:21.820348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:21.820577 (+ 229us) server_negotiation.cc:407] Connection header received
0925 09:19:21.820822 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:21.820830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:21.820968 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:21.821091 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:21.822064 (+ 973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42222: BlockingRecv error: recv got EOF from 127.0.0.1:42222 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 09:19:22.077656 25941 catalog_manager.cc:1259] Loaded cluster ID: c80afabd6d534d8898a409e1310b122c
I20260925 09:19:22.077775 25941 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782: loading cluster ID for follower catalog manager: success
I20260925 09:19:22.080336 25941 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782: acquiring CA information for follower catalog manager: success
I20260925 09:19:22.082242 25941 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 28eea4b2a4184275b7119ecd438ca782: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 09:19:22.347661 26007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:22.340070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.3.19.125:34067)
0925 09:19:22.341033 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:22.341038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:22.341056 (+ 18us) server_negotiation.cc:407] Connection header received
0925 09:19:22.341119 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:22.341123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:22.341348 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:22.341463 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:19:22.342017 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:22.342914 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:22.344961 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:22.345478 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:22.345586 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:22.346268 (+ 682us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:19:22.346872 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:19:22.347000 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:22.347172 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 09:19:22.347244 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":663,"threads_started":1}
I20260925 09:19:22.376223 26013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:22.354798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49224 (local address 127.3.19.124:35987)
0925 09:19:22.369954 (+ 15156us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:22.369959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:22.369980 (+ 21us) server_negotiation.cc:407] Connection header received
0925 09:19:22.370083 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:22.370087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:22.370249 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:22.370422 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:19:22.371028 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:22.371776 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:22.373853 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:22.374256 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:22.374349 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:22.374989 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:19:22.375518 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:19:22.375655 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:22.375810 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 09:19:22.375880 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15028,"thread_start_us":15706,"threads_started":1}
I20260925 09:19:22.399562 26007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:22.384094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.3.19.125:34067)
0925 09:19:22.384302 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:22.384311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:22.391471 (+ 7160us) server_negotiation.cc:407] Connection header received
0925 09:19:22.391905 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:22.391914 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:22.392047 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:22.392130 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:19:22.393091 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:22.394492 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:22.397260 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:22.397903 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:22.398035 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:22.398692 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:19:22.398823 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:19:22.398883 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:22.399128 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 09:19:22.399237 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 09:19:22.404053 26013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:22.384469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.3.19.124:35987)
0925 09:19:22.384685 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:22.384693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:22.396998 (+ 12305us) server_negotiation.cc:407] Connection header received
0925 09:19:22.397484 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:22.397490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:22.397593 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:22.397654 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:19:22.398385 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:22.399488 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:22.401938 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:22.402621 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:22.402756 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:22.403260 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:19:22.403400 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:19:22.403498 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:22.403694 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 09:19:22.403776 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 09:19:22.406327 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:22.384323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.3.19.126:43111)
0925 09:19:22.394004 (+ 9681us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:22.394010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:22.398920 (+ 4910us) server_negotiation.cc:407] Connection header received
0925 09:19:22.399457 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:22.399463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:22.399687 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:22.399842 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:19:22.400712 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:22.401819 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:22.403637 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:22.404281 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:22.404398 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:22.405261 (+ 863us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:19:22.405684 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:19:22.405769 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:22.405949 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 09:19:22.406018 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9555,"thread_start_us":9454,"threads_started":1}
I20260925 09:19:22.411372 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 25658
I20260925 09:19:22.412443 25720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:19:22.528810 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 25658
W20260925 09:19:22.550740 25825 connection.cc:535] server connection from 127.0.0.1:49192 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 09:19:22.551045 25751 connection.cc:535] server connection from 127.0.0.1:42112 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 09:19:22.551256 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 25732
I20260925 09:19:22.552364 25794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:19:22.667217 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 25732
I20260925 09:19:22.689136 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 25806
I20260925 09:19:22.690431 25868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:19:22.799544 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 25806
2026-09-25T09:19:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5090 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 09:19:22 dist-test-slave-l6qh krb5kdc[26028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 09:19:22 dist-test-slave-l6qh krb5kdc[26028](info): set up 2 sockets
Sep 25 09:19:22 dist-test-slave-l6qh krb5kdc[26028](info): commencing operation
krb5kdc: starting...
W20260925 09:19:23.934899 3149 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.006s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/test-user.keytab.
Sep 25 09:19:24 dist-test-slave-l6qh krb5kdc[26028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T09:19:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T09:19:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
Entry for principal kudu/127.3.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab.
I20260925 09:19:24.221155 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:42489
--webserver_interface=127.3.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:42489,127.3.19.125:33195,127.3.19.124:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 09:19:24.553874 26041 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 09:19:24.554193 26041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:24.554319 26041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:24.554425 26041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:24.561786 26041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:24.561874 26041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:24.561973 26041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:24.562072 26041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:24.562186 26041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:24.572084 26041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:42489,127.3.19.125:33195,127.3.19.124:34373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.126.keytab
--principal=kudu/127.3.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.126:42489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:24.573539 26041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:24.574887 26041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:19:24.587338 26049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:24.587272 26047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:24.587347 26046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:24.588280 26041 server_base.cc:1034] running on GCE node
I20260925 09:19:24.589125 26041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:24.590596 26041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:24.591815 26041 hybrid_clock.cc:648] HybridClock initialized: now 1790327964591764 us; error 23 us; skew 500 ppm
I20260925 09:19:24.592186 26041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:24 dist-test-slave-l6qh krb5kdc[26028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327964, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:24.595381 26041 init.cc:377] Logged in from keytab as kudu/127.3.19.126@KRBTEST.COM (short username kudu)
I20260925 09:19:24.597203 26041 webserver.cc:460] Webserver started at http://127.3.19.126:41583/ using document root <none> and password file <none>
I20260925 09:19:24.597913 26041 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:24.598011 26041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:24.598277 26041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:24.601214 26041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/data/instance:
uuid: "c6f83f9da3b147d18a2fae2896fa978f"
format_stamp: "Formatted at 2026-09-25 09:19:24 on dist-test-slave-l6qh"
server_key: "acb7e79e25b8c6341306a1c766d16cad"
server_key_iv: "d660e7f5d99e68ce603c8b4053c43246"
server_key_version: "encryptionkey@0"
I20260925 09:19:24.602062 26041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/wal/instance:
uuid: "c6f83f9da3b147d18a2fae2896fa978f"
format_stamp: "Formatted at 2026-09-25 09:19:24 on dist-test-slave-l6qh"
server_key: "acb7e79e25b8c6341306a1c766d16cad"
server_key_iv: "d660e7f5d99e68ce603c8b4053c43246"
server_key_version: "encryptionkey@0"
I20260925 09:19:24.607431 26041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 09:19:24.611084 26056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:24.612598 26041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 09:19:24.612753 26041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/wal
uuid: "c6f83f9da3b147d18a2fae2896fa978f"
format_stamp: "Formatted at 2026-09-25 09:19:24 on dist-test-slave-l6qh"
server_key: "acb7e79e25b8c6341306a1c766d16cad"
server_key_iv: "d660e7f5d99e68ce603c8b4053c43246"
server_key_version: "encryptionkey@0"
I20260925 09:19:24.612941 26041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:24.631995 26041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:24.636351 26041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:24.636596 26041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:24.656509 26041 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.126:42489
I20260925 09:19:24.656556 26107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.126:42489 every 8 connection(s)
I20260925 09:19:24.658030 26041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/data/info.pb
I20260925 09:19:24.661891 26108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:24.668102 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 26041
I20260925 09:19:24.668304 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-0/wal/instance
I20260925 09:19:24.668823 3149 external_mini_cluster.cc:1428] Setting key 869dcdb40f92ec1e392c8bed4cfb4687
I20260925 09:19:24.668603 26108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:19:24 dist-test-slave-l6qh krb5kdc[26028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327964, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:19:24.691498 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:24.672007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.3.19.126:42489)
0925 09:19:24.672852 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:24.672870 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:24.673370 (+ 500us) server_negotiation.cc:407] Connection header received
0925 09:19:24.675342 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:24.675348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:24.675584 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:24.675792 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:24.677081 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:24.678336 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:24.679733 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:24.680269 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:24.680474 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:24.684084 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:24.684092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:24.684103 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:24.684130 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:24.686948 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:24.687670 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:24.687678 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:24.687681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:24.687823 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:24.688220 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:24.688228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:24.688232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:24.688834 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:24.689058 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:24.689637 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 09:19:24.689981 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":597,"thread_start_us":382,"threads_started":1}
I20260925 09:19:24.691478 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:24.672007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:42489 (local address 127.0.0.1:42618)
0925 09:19:24.673162 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:24.673198 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:24.674973 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:24.675814 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:24.675820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:24.676521 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:24.676860 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:24.676876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:24.678500 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:24.678505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:24.679592 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:24.679603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:24.680502 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:24.680509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:24.680693 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:24.681404 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:24.681426 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:24.683849 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:24.687125 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:24.687136 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:24.687147 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:24.687510 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:24.687940 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:24.687947 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:24.687952 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:24.688114 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:24.688983 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:24.688991 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:24.689211 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 09:19:24.689521 (+ 310us) client_negotiation.cc:240] Negotiation successful
0925 09:19:24.690108 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":145,"thread_start_us":735,"threads_started":1}
I20260925 09:19:24.694895 26108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } has no permanent_uuid. Determining permanent_uuid...
I20260925 09:19:24.695959 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:24.695591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:33195 (local address 127.0.0.1:38644)
0925 09:19:24.695752 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:24.695851 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.3.19.125@KRBTEST.COM; defaulting to no policy
W20260925 09:19:24.696198 26057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
I20260925 09:19:24.697778 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:24.697420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:33195 (local address 127.0.0.1:38660)
0925 09:19:24.697592 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:24.697658 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 09:19:24.697984 26108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:33195: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.125.keytab.
Entry for principal kudu/127.3.19.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.125.keytab.
I20260925 09:19:24.732265 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.125:33195
--webserver_interface=127.3.19.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.125.keytab
--principal=kudu/127.3.19.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:42489,127.3.19.125:33195,127.3.19.124:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:19:24.747332 26108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } attempt: 1
I20260925 09:19:24.748643 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:24.748054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:33195 (local address 127.0.0.1:38662)
0925 09:19:24.748319 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:24.748441 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 09:19:24.749850 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:24.749402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:33195 (local address 127.0.0.1:38664)
0925 09:19:24.749578 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:24.749670 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 09:19:24.750061 26108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:33195: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
I20260925 09:19:24.818347 26108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } attempt: 2
I20260925 09:19:24.819609 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:24.819024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:33195 (local address 127.0.0.1:38680)
0925 09:19:24.819309 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:24.819425 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 09:19:24.820698 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:24.820298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:33195 (local address 127.0.0.1:38682)
0925 09:19:24.820446 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:24.820546 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 09:19:24.820890 26108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:33195: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
I20260925 09:19:24.912204 26108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } attempt: 3
I20260925 09:19:24.913625 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:24.912958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:33195 (local address 127.0.0.1:38698)
0925 09:19:24.913274 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:24.913413 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 09:19:24.914862 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:24.914376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:33195 (local address 127.0.0.1:38712)
0925 09:19:24.914559 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:24.914665 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 09:19:24.915082 26108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:33195: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
I20260925 09:19:25.058375 26108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } attempt: 4
I20260925 09:19:25.059661 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.059049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:33195 (local address 127.0.0.1:38728)
0925 09:19:25.059386 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.059496 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260925 09:19:25.060729 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.060320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:33195 (local address 127.0.0.1:38734)
0925 09:19:25.060471 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.060570 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 09:19:25.060947 26108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.125:33195: Network error: Client connection negotiation failed: client connection to 127.3.19.125:33195: connect: Connection refused (error 111)
W20260925 09:19:25.071208 26115 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 09:19:25.071569 26115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:25.071672 26115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:25.071813 26115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:25.079109 26115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:25.079206 26115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:25.079340 26115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:25.079424 26115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:25.079546 26115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:25.089074 26115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:42489,127.3.19.125:33195,127.3.19.124:34373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.125.keytab
--principal=kudu/127.3.19.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.125:33195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.19.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:25.090385 26115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:25.091694 26115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 09:19:25.103945 26120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:25.103888 26121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:25.103924 26123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:25.104732 26115 server_base.cc:1034] running on GCE node
I20260925 09:19:25.105623 26115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:25.107057 26115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:25.108316 26115 hybrid_clock.cc:648] HybridClock initialized: now 1790327965108237 us; error 53 us; skew 500 ppm
I20260925 09:19:25.108655 26115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:25 dist-test-slave-l6qh krb5kdc[26028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:25.111964 26115 init.cc:377] Logged in from keytab as kudu/127.3.19.125@KRBTEST.COM (short username kudu)
I20260925 09:19:25.113632 26115 webserver.cc:460] Webserver started at http://127.3.19.125:36953/ using document root <none> and password file <none>
I20260925 09:19:25.114413 26115 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:25.114575 26115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:25.114918 26115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:25.118273 26115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/data/instance:
uuid: "5285199b039f43a2adcf28d86d3bb964"
format_stamp: "Formatted at 2026-09-25 09:19:25 on dist-test-slave-l6qh"
server_key: "437b072d2ca2635587098ada2a3661e3"
server_key_iv: "720b14b6342ebcd77274b62edd58b86a"
server_key_version: "encryptionkey@0"
I20260925 09:19:25.119374 26115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/wal/instance:
uuid: "5285199b039f43a2adcf28d86d3bb964"
format_stamp: "Formatted at 2026-09-25 09:19:25 on dist-test-slave-l6qh"
server_key: "437b072d2ca2635587098ada2a3661e3"
server_key_iv: "720b14b6342ebcd77274b62edd58b86a"
server_key_version: "encryptionkey@0"
I20260925 09:19:25.124621 26115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 09:19:25.128237 26130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:25.129828 26115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 09:19:25.130044 26115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/wal
uuid: "5285199b039f43a2adcf28d86d3bb964"
format_stamp: "Formatted at 2026-09-25 09:19:25 on dist-test-slave-l6qh"
server_key: "437b072d2ca2635587098ada2a3661e3"
server_key_iv: "720b14b6342ebcd77274b62edd58b86a"
server_key_version: "encryptionkey@0"
I20260925 09:19:25.130290 26115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:25.145653 26115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:25.150074 26115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:25.150380 26115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:25.170006 26115 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.125:33195
I20260925 09:19:25.170040 26181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.125:33195 every 8 connection(s)
I20260925 09:19:25.171406 26115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/data/info.pb
I20260925 09:19:25.175117 26182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:25.179020 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 26115
I20260925 09:19:25.179219 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-1/wal/instance
I20260925 09:19:25.179699 3149 external_mini_cluster.cc:1428] Setting key 69512d070688497fad23a0f0001c4bc9
I20260925 09:19:25.182535 26182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:19:25 dist-test-slave-l6qh krb5kdc[26028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.125@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:19:25.205535 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.185175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.3.19.126:42489)
0925 09:19:25.185413 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.185420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.186610 (+ 1190us) server_negotiation.cc:407] Connection header received
0925 09:19:25.188485 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.188494 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.188587 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.188708 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.190227 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.191007 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.193547 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.194031 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.194163 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.198810 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.198816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.198819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.198852 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.201799 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.202765 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.202768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.202771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.202885 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.203465 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.203473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.203476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.203832 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.203976 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.205037 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.205218 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 09:19:25.206993 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.185605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:42489 (local address 127.0.0.1:42632)
0925 09:19:25.186413 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.186460 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:25.188212 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:25.188896 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:25.188903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:25.189487 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:25.190046 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.190063 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.191178 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.191186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.193351 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.193369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.194284 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.194293 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.194645 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.195423 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:25.195463 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:25.198548 (+ 3085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:25.201938 (+ 3390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.201950 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.201971 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.202642 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.202989 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.202997 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.203004 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.203258 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.203983 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:25.203993 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:25.204551 (+ 558us) client_negotiation.cc:769] Sending connection context
0925 09:19:25.204901 (+ 350us) client_negotiation.cc:240] Negotiation successful
0925 09:19:25.205517 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":360,"threads_started":1}
WARNING: no policy specified for kudu/127.3.19.124@KRBTEST.COM; defaulting to no policy
I20260925 09:19:25.208858 26182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.19.124@KRBTEST.COM" created.
Sep 25 09:19:25 dist-test-slave-l6qh krb5kdc[26028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.125@KRBTEST.COM for kudu/127.3.19.125@KRBTEST.COM
I20260925 09:19:25.225798 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.209704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38736 (local address 127.3.19.125:33195)
0925 09:19:25.210489 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.210499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.210531 (+ 32us) server_negotiation.cc:407] Connection header received
0925 09:19:25.210587 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.210598 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.210840 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.210957 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.211931 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.212956 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.214831 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.215364 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.215461 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.219219 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.219236 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.219257 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.219305 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.222683 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.223515 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.223523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.223525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.223635 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.224044 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.224047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.224049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.224393 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.224567 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.225081 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.225457 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":595,"thread_start_us":404,"threads_started":1}
I20260925 09:19:25.225884 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.209630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:33195 (local address 127.0.0.1:38736)
0925 09:19:25.209874 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.209890 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:25.209973 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:25.211027 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:25.211033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:25.211530 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:25.211754 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.211769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.213107 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.213115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.214654 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.214671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.215533 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.215545 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.215772 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.216449 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:25.216475 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:25.218958 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:25.222816 (+ 3858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.222824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.222831 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.223383 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.223767 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.223775 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.223781 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.223941 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.224526 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:25.224532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:25.224716 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 09:19:25.224901 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 09:19:25.225337 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 09:19:25.228438 26182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } has no permanent_uuid. Determining permanent_uuid...
I20260925 09:19:25.229684 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.229186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55012)
0925 09:19:25.229417 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.229539 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260925 09:19:25.229950 26134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
I20260925 09:19:25.231619 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.231242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55014)
0925 09:19:25.231401 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.231519 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 09:19:25.231834 26182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:34373: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.19.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.124.keytab.
Entry for principal kudu/127.3.19.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.124.keytab.
I20260925 09:19:25.245965 3149 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
/tmp/dist-test-taskCZfsc6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.124:34373
--webserver_interface=127.3.19.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.124.keytab
--principal=kudu/127.3.19.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.19.84:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.19.126:42489,127.3.19.125:33195,127.3.19.124:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 09:19:25.281162 26182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } attempt: 1
I20260925 09:19:25.282348 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.281882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55030)
0925 09:19:25.282086 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.282198 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 09:19:25.283384 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.283057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55042)
0925 09:19:25.283202 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.283285 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 09:19:25.283645 26182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:34373: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
I20260925 09:19:25.351992 26182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } attempt: 2
I20260925 09:19:25.353083 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.352649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55054)
0925 09:19:25.352837 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.352929 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 09:19:25.354182 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.353864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55068)
0925 09:19:25.354009 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.354078 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 09:19:25.354399 26182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:34373: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
I20260925 09:19:25.360198 26108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } attempt: 5
Sep 25 09:19:25 dist-test-slave-l6qh krb5kdc[26028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327964, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for kudu/127.3.19.125@KRBTEST.COM
I20260925 09:19:25.374230 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.360855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:33195 (local address 127.0.0.1:38752)
0925 09:19:25.361075 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.361096 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:25.361254 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:25.361636 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:25.361640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:25.362059 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:25.362230 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.362238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.363197 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.363203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.364069 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.364080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.364847 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.364853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.365081 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.365723 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:25.365740 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:25.367695 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:25.371889 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.371893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.371895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.372260 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.372715 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.372718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.372720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.372848 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.373561 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:25.373564 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:25.373689 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 09:19:25.373798 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 09:19:25.373984 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 09:19:25.374543 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.360993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.3.19.125:33195)
0925 09:19:25.361171 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.361176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.361196 (+ 20us) server_negotiation.cc:407] Connection header received
0925 09:19:25.361406 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.361410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.361476 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.361603 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.362377 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.363069 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.364209 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.364698 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.364824 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.367905 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.367912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.367918 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.367973 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.371735 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.372389 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.372398 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.372404 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.372591 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.372960 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.372967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.372971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.373440 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.373619 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.373962 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.374192 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 09:19:25.375554 26108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } has no permanent_uuid. Determining permanent_uuid...
I20260925 09:19:25.376506 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.376142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55084)
0925 09:19:25.376294 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.376368 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 09:19:25.377445 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.377103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55100)
0925 09:19:25.377249 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.377351 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 09:19:25.377679 26108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:34373: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
I20260925 09:19:25.428967 26108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } attempt: 1
I20260925 09:19:25.430379 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.429725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55108)
0925 09:19:25.429983 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.430123 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 09:19:25.431490 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.431096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55110)
0925 09:19:25.431256 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.431347 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 09:19:25.431751 26108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:34373: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
I20260925 09:19:25.445622 26182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } attempt: 3
I20260925 09:19:25.446729 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.446243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55122)
0925 09:19:25.446439 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.446567 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 09:19:25.447810 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.447414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55126)
0925 09:19:25.447638 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.447708 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260925 09:19:25.447983 26182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:34373: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
I20260925 09:19:25.500012 26108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } attempt: 2
I20260925 09:19:25.501349 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.500712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55132)
0925 09:19:25.500962 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.501100 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 09:19:25.502542 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.502095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55144)
0925 09:19:25.502285 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.502380 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 09:19:25.502851 26108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:34373: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
W20260925 09:19:25.579836 26189 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 09:19:25.580183 26189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 09:19:25.580296 26189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 09:19:25.580425 26189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 09:19:25.587811 26189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 09:19:25.587893 26189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 09:19:25.588033 26189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 09:19:25.588106 26189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 09:19:25.588210 26189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 09:19:25.591177 26182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } attempt: 4
I20260925 09:19:25.592434 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.591851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55152)
0925 09:19:25.592095 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.592235 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 09:19:25.593595 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.593152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55166)
0925 09:19:25.593362 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.593448 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 09:19:25.593822 26182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:34373: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
I20260925 09:19:25.598790 26189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.19.84:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.19.126:42489,127.3.19.125:33195,127.3.19.124:34373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/krb5kdc/kudu_127.3.19.124.keytab
--principal=kudu/127.3.19.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.19.124:34373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.19.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 07:45:44 UTC on ccycloud.quasar-yomzrf.root.comops.site
build id 4671939
ASAN enabled
I20260925 09:19:25.600064 26189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 09:19:25.601445 26189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 09:19:25.609107 26108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } attempt: 3
I20260925 09:19:25.610327 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.609795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55182)
0925 09:19:25.610035 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.610135 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 09:19:25.611575 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.611027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55190)
0925 09:19:25.611290 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.611403 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260925 09:19:25.611956 26108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.19.124:34373: Network error: Client connection negotiation failed: client connection to 127.3.19.124:34373: connect: Connection refused (error 111)
W20260925 09:19:25.613575 26194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:25.613646 26195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 09:19:25.613704 26197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 09:19:25.614322 26189 server_base.cc:1034] running on GCE node
I20260925 09:19:25.615126 26189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 09:19:25.616724 26189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 09:19:25.617956 26189 hybrid_clock.cc:648] HybridClock initialized: now 1790327965617899 us; error 28 us; skew 500 ppm
I20260925 09:19:25.618310 26189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 09:19:25 dist-test-slave-l6qh krb5kdc[26028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 09:19:25.624379 26189 init.cc:377] Logged in from keytab as kudu/127.3.19.124@KRBTEST.COM (short username kudu)
I20260925 09:19:25.626060 26189 webserver.cc:460] Webserver started at http://127.3.19.124:45379/ using document root <none> and password file <none>
I20260925 09:19:25.626770 26189 fs_manager.cc:359] Metadata directory not provided
I20260925 09:19:25.626895 26189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 09:19:25.627208 26189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 09:19:25.630264 26189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/data/instance:
uuid: "9e64f08e26e44935a398c82c3c9de672"
format_stamp: "Formatted at 2026-09-25 09:19:25 on dist-test-slave-l6qh"
server_key: "5ef64c911b8206d94be8ad70b7a03ced"
server_key_iv: "cf844d6d7ec8eccee086df270337bdf1"
server_key_version: "encryptionkey@0"
I20260925 09:19:25.631032 26189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/wal/instance:
uuid: "9e64f08e26e44935a398c82c3c9de672"
format_stamp: "Formatted at 2026-09-25 09:19:25 on dist-test-slave-l6qh"
server_key: "5ef64c911b8206d94be8ad70b7a03ced"
server_key_iv: "cf844d6d7ec8eccee086df270337bdf1"
server_key_version: "encryptionkey@0"
I20260925 09:19:25.636116 26189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 09:19:25.639504 26204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 09:19:25.640776 26189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 09:19:25.640944 26189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/data,/tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/wal
uuid: "9e64f08e26e44935a398c82c3c9de672"
format_stamp: "Formatted at 2026-09-25 09:19:25 on dist-test-slave-l6qh"
server_key: "5ef64c911b8206d94be8ad70b7a03ced"
server_key_iv: "cf844d6d7ec8eccee086df270337bdf1"
server_key_version: "encryptionkey@0"
I20260925 09:19:25.641148 26189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 09:19:25.659607 26189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 09:19:25.663676 26189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 09:19:25.663933 26189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 09:19:25.682365 26189 rpc_server.cc:307] RPC server started. Bound to: 127.3.19.124:34373
I20260925 09:19:25.682384 26255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.19.124:34373 every 8 connection(s)
I20260925 09:19:25.683689 26189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/data/info.pb
I20260925 09:19:25.687249 26256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 09:19:25.691051 3149 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu as pid 26189
I20260925 09:19:25.691254 3149 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCZfsc6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790327600440376-3149-0/minicluster-data/master-2/wal/instance
I20260925 09:19:25.691783 3149 external_mini_cluster.cc:1428] Setting key 74dc66bb31a82cf361c2875a9d8a16c7
I20260925 09:19:25.694319 26256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:19:25 dist-test-slave-l6qh krb5kdc[26028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.124@KRBTEST.COM
Sep 25 09:19:25 dist-test-slave-l6qh krb5kdc[26028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.124@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:19:25.714279 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.698598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.3.19.126:42489)
0925 09:19:25.698852 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.698859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.699369 (+ 510us) server_negotiation.cc:407] Connection header received
0925 09:19:25.699569 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.699577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.699674 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.699795 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.700967 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.701770 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.703339 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.703842 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.704005 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.708090 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.708095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.708098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.708129 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.711197 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.712121 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.712125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.712127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.712264 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.712701 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.712704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.712706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.713023 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.713174 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.713830 (+ 656us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.713999 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 09:19:25.715215 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.698554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.126:42489 (local address 127.0.0.1:42644)
0925 09:19:25.699213 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.699270 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:25.699419 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:25.699838 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:25.699843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:25.700484 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:25.700791 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.700803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.701932 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.701937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.703177 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.703189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.704108 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.704117 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.704287 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.704939 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:25.704967 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:25.707882 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:25.711362 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.711371 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.711382 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.712017 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.712369 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.712374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.712377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.712589 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.713185 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:25.713194 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:25.713472 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 09:19:25.713675 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 09:19:25.714061 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":93,"spinlock_wait_cycles":16256,"thread_start_us":286,"threads_started":1}
I20260925 09:19:25.715240 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.693668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.3.19.124:34373)
0925 09:19:25.694487 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.694507 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.694562 (+ 55us) server_negotiation.cc:407] Connection header received
0925 09:19:25.696629 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.696659 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.697169 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.697599 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.699311 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.700893 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.702457 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.703158 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.703339 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.707546 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.707562 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.707577 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.707617 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.710558 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.711370 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.711378 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.711382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.711622 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.712017 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.712023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.712027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.712733 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.712977 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.713479 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.713947 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"spinlock_wait_cycles":15744,"thread_start_us":340,"threads_started":1}
I20260925 09:19:25.717132 26256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 09:19:25 dist-test-slave-l6qh krb5kdc[26028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.124@KRBTEST.COM for kudu/127.3.19.125@KRBTEST.COM
Sep 25 09:19:25 dist-test-slave-l6qh krb5kdc[26028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.126@KRBTEST.COM
I20260925 09:19:25.730469 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.717924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.125:33195 (local address 127.0.0.1:38768)
0925 09:19:25.718180 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.718198 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:25.718298 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:25.718692 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:25.718697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:25.719083 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:25.719232 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.719242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.720246 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.720254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.721203 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.721215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.722001 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.722007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.722197 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.722762 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:25.722779 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:25.724724 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:25.728052 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.728058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.728062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.728565 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.728915 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.728921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.728926 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.729117 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.729723 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:25.729730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:25.729884 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 09:19:25.730052 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 09:19:25.730242 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260925 09:19:25.730785 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.717924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.3.19.125:33195)
0925 09:19:25.718158 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.718164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.718252 (+ 88us) server_negotiation.cc:407] Connection header received
0925 09:19:25.718451 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.718458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.718548 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.718660 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.719390 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.720103 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.721401 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.721849 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.721996 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.724920 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.724926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.724931 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.724978 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.727927 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.728672 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.728676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.728678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.728808 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.729302 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.729306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.729327 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.729596 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.729714 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.730165 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.730389 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 09:19:25.731869 26256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } has no permanent_uuid. Determining permanent_uuid...
I20260925 09:19:25.733003 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.718484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.3.19.126:42489)
0925 09:19:25.718646 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.718651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.718713 (+ 62us) server_negotiation.cc:407] Connection header received
0925 09:19:25.718913 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.718918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.718990 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.719084 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.720183 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.720909 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.722153 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.722744 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.722876 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.726956 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.726963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.726968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.726997 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.729897 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.730824 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.730827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.730829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.731001 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.731392 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.731397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.731401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.731768 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.732053 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.732323 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.732579 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 09:19:25 dist-test-slave-l6qh krb5kdc[26028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.124@KRBTEST.COM for kudu/127.3.19.124@KRBTEST.COM
Sep 25 09:19:25 dist-test-slave-l6qh krb5kdc[26028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.19.125@KRBTEST.COM
I20260925 09:19:25.745482 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.732733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.3.19.124:34373)
0925 09:19:25.732893 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.732898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.733017 (+ 119us) server_negotiation.cc:407] Connection header received
0925 09:19:25.733204 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.733210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.733301 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.733364 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.734208 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.735342 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.736607 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.737067 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.737179 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.740249 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.740253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.740256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.740282 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.742780 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.743433 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.743440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.743445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.743559 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.743888 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.743895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.743899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.744259 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.744400 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.744831 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.745095 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 09:19:25.745643 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.732730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55204)
0925 09:19:25.732936 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.732954 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:25.733071 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:25.733417 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:25.733422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:25.733844 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:25.734081 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.734096 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.735589 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.735593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.736437 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.736447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.737213 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.737219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.737468 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.738095 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:25.738110 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:25.740079 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:25.742905 (+ 2826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.742912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.742916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.743330 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.743659 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.743666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.743671 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.743794 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.744390 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:25.744394 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:25.744537 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 09:19:25.744720 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 09:19:25.745324 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":132}
I20260925 09:19:25.749825 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.734751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.3.19.125:33195)
0925 09:19:25.734904 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.734910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.734929 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:19:25.735048 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.735051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.735101 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.735171 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.736603 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.737436 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.738841 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.739378 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.739473 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.744070 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.744076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.744080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.744117 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.747428 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.748181 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.748184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.748186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.748307 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.748696 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.748699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.748701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.748928 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.749122 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.749425 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.749594 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 09:19:25.751691 26256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672: Bootstrap starting.
I20260925 09:19:25.754222 3149 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 09:19:25.755532 26256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:25.756788 26256 log.cc:824] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:25.759618 26256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672: No bootstrap required, opened a new log
I20260925 09:19:25.757179 26083 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c6f83f9da3b147d18a2fae2896fa978f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42656
I20260925 09:19:25.767028 26256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } }
I20260925 09:19:25.767372 26256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e64f08e26e44935a398c82c3c9de672, State: Initialized, Role: FOLLOWER
I20260925 09:19:25.767839 26256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } }
I20260925 09:19:25.768711 26262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } } }
I20260925 09:19:25.769052 26262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:19:25.770061 26256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 09:19:25.779433 26273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 09:19:25.779606 26273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 09:19:25.777489 26189 master_runner.cc:186] Error getting master registration for 127.3.19.126:42489: OK, instance_id { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" instance_seqno: 1790327964650895 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 09:19:25.789207 26189 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 09:19:25.789180 26108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } attempt: 4
Sep 25 09:19:25 dist-test-slave-l6qh krb5kdc[26028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327964, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.126@KRBTEST.COM for kudu/127.3.19.124@KRBTEST.COM
I20260925 09:19:25.806147 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.789946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.3.19.124:34373)
0925 09:19:25.790487 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.790495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.790513 (+ 18us) server_negotiation.cc:407] Connection header received
0925 09:19:25.790570 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.790576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.790656 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.790742 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.791773 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.792892 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.794808 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.795667 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.795784 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.799834 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.799839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.799842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.799871 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.803405 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.804093 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.804099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.804102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.804227 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.804598 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.804604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.804607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.804950 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.805123 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.805569 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.805777 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388}
I20260925 09:19:25.806144 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.789908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55220)
0925 09:19:25.790119 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.790139 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:25.790257 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:25.790835 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:25.790840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:25.791361 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:25.791611 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.791626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.793108 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.793113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.794651 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.794667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.795978 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.795986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.796210 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.796917 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:25.796939 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:25.799663 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:25.803536 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.803541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.803545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.803955 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.804335 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.804340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.804344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.804491 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.805077 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:25.805084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:25.805284 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 09:19:25.805440 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 09:19:25.805756 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 09:19:25.810459 26108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f: Bootstrap starting.
I20260925 09:19:25.813550 26108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:25.814550 26108 log.cc:824] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:25.817497 26108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f: No bootstrap required, opened a new log
I20260925 09:19:25.822608 26108 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } }
I20260925 09:19:25.822919 26108 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f83f9da3b147d18a2fae2896fa978f, State: Initialized, Role: FOLLOWER
I20260925 09:19:25.823341 26108 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } }
I20260925 09:19:25.824388 26276 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } } }
I20260925 09:19:25.824815 26276 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:19:25.825587 26108 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:25.825719 26083 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 09:19:25.826004 26083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:25.830070 26083 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } }
I20260925 09:19:25.833158 26231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6f83f9da3b147d18a2fae2896fa978f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9e64f08e26e44935a398c82c3c9de672"
I20260925 09:19:25.834787 26231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:25.838822 26231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f83f9da3b147d18a2fae2896fa978f in term 1.
I20260925 09:19:25.839700 26060 leader_election.cc:302] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e64f08e26e44935a398c82c3c9de672, c6f83f9da3b147d18a2fae2896fa978f; no voters:
I20260925 09:19:25.840265 26276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 09:19:25.845531 26276 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [term 1 LEADER]: Becoming Leader. State: Replica: c6f83f9da3b147d18a2fae2896fa978f, State: Running, Role: LEADER
I20260925 09:19:25.846319 26276 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } }
I20260925 09:19:25.849184 26083 leader_election.cc:288] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [CANDIDATE]: Term 1 election: Requested vote from peers 5285199b039f43a2adcf28d86d3bb964 (127.3.19.125:33195), 9e64f08e26e44935a398c82c3c9de672 (127.3.19.124:34373)
W20260925 09:19:25.857023 26287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 09:19:25.857195 26287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 09:19:25.858289 26289 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6f83f9da3b147d18a2fae2896fa978f. Latest consensus state: current_term: 1 leader_uuid: "c6f83f9da3b147d18a2fae2896fa978f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } } }
I20260925 09:19:25.858798 26289 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:25.836946 26041 master_runner.cc:186] Error getting master registration for 127.3.19.125:33195: OK, instance_id { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" instance_seqno: 1790327965164607 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 09:19:25.861724 26041 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 09:19:25.873881 26293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 09:19:25.876291 26293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 09:19:25.889416 26062 tablet.cc:2369] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:19:25.893008 26182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } attempt: 5
I20260925 09:19:25.898953 26231 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672 [term 1 FOLLOWER]: Refusing update from remote peer c6f83f9da3b147d18a2fae2896fa978f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:19:25.900066 26276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 09:19:25.929908 26210 tablet.cc:2369] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 09:19:25.934689 26262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6f83f9da3b147d18a2fae2896fa978f. Latest consensus state: current_term: 1 leader_uuid: "c6f83f9da3b147d18a2fae2896fa978f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } } }
I20260925 09:19:25.935024 26262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:19:25.937901 26296 mvcc.cc:205] Tried to move back new op lower bound from 7333183348272144384 to 7333183348113317888. Current Snapshot: MvccSnapshot[applied={T|T < 7333183348272144384}]
Sep 25 09:19:25 dist-test-slave-l6qh krb5kdc[26028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790327965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.19.125@KRBTEST.COM for kudu/127.3.19.124@KRBTEST.COM
I20260925 09:19:25.954046 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.882419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42662 (local address 127.3.19.126:42489)
0925 09:19:25.882617 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.882625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.884269 (+ 1644us) server_negotiation.cc:407] Connection header received
0925 09:19:25.888328 (+ 4059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.888336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.888419 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.888515 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.895489 (+ 6974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.896671 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.902773 (+ 6102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.903447 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.903559 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.924134 (+ 20575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.924143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.924148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.924197 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.928350 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.932444 (+ 4094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.932453 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.932458 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.932655 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.952003 (+ 19348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.952009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.952013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.952444 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.952717 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.953370 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.953652 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 09:19:25.957563 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.884971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38798 (local address 127.3.19.125:33195)
0925 09:19:25.885156 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.885162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.892661 (+ 7499us) server_negotiation.cc:407] Connection header received
0925 09:19:25.892938 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.892944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.893043 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.893127 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.900290 (+ 7163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.901494 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.909497 (+ 8003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.910239 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.910358 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.932967 (+ 22609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.932976 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.932981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.933032 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.938359 (+ 5327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.955147 (+ 16788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.955153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.955156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.955310 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.955778 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.955783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.955786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.956133 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.956346 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.956940 (+ 594us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.957177 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 09:19:25.971174 26276 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6f83f9da3b147d18a2fae2896fa978f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } } }
I20260925 09:19:25.971496 26276 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:25.980067 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.893553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.19.124:34373 (local address 127.0.0.1:55242)
0925 09:19:25.893783 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 09:19:25.893799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 09:19:25.893894 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 09:19:25.936317 (+ 42423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 09:19:25.936323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 09:19:25.937677 (+ 1354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 09:19:25.937929 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.937997 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.942940 (+ 4943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.942948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.944357 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 09:19:25.944377 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.945738 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.945748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 09:19:25.946041 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.946944 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 09:19:25.946977 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 09:19:25.950086 (+ 3109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 09:19:25.971606 (+ 21520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.971612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.971615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.972161 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.973267 (+ 1106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 09:19:25.973272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 09:19:25.973276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 09:19:25.973459 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 09:19:25.978959 (+ 5500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 09:19:25.978964 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 09:19:25.979137 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 09:19:25.979297 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 09:19:25.979670 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260925 09:19:25.982837 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.894153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.3.19.124:34373)
0925 09:19:25.935711 (+ 41558us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.935716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.935740 (+ 24us) server_negotiation.cc:407] Connection header received
0925 09:19:25.935791 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.935795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.935880 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.935994 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.938664 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.942767 (+ 4103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.944652 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.945502 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.950268 (+ 4766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.950578 (+ 310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.950585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.950590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.950627 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.966795 (+ 16168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.972384 (+ 5589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.972389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.972393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.972576 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.973611 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.973616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.973620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.974037 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.980728 (+ 6691us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.980857 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.981090 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41425,"thread_start_us":42025,"threads_started":1}
I20260925 09:19:25.985368 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:25.891176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55228 (local address 127.3.19.124:34373)
0925 09:19:25.891357 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:25.891363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:25.909273 (+ 17910us) server_negotiation.cc:407] Connection header received
0925 09:19:25.911308 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:25.911316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:25.911414 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:25.911521 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:25.930766 (+ 19245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.932782 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.960767 (+ 27985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:25.961572 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:25.961679 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:25.974233 (+ 12554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 09:19:25.974243 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 09:19:25.974248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 09:19:25.974283 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 09:19:25.978100 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.981327 (+ 3227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.981332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.981338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.981498 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 09:19:25.983176 (+ 1678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 09:19:25.983182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 09:19:25.983185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 09:19:25.983533 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 09:19:25.983785 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:25.984748 (+ 963us) server_negotiation.cc:299] Negotiation successful
0925 09:19:25.984989 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 09:19:25.987313 26293 catalog_manager.cc:1347] Generated new cluster ID: 512a606894234861ba761184caea90ec
I20260925 09:19:25.987412 26293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 09:19:25.990146 26182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964: Bootstrap starting.
I20260925 09:19:25.996764 26182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964: Neither blocks nor log segments found. Creating new log.
I20260925 09:19:25.999200 26182 log.cc:824] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964: Log is configured to *not* fsync() on all Append() calls
I20260925 09:19:26.000589 26262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6f83f9da3b147d18a2fae2896fa978f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } } }
I20260925 09:19:26.000874 26262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:19:26.011230 26293 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 09:19:26.014536 26293 catalog_manager.cc:1504] Loading token signing keys...
I20260925 09:19:26.022258 26182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964: No bootstrap required, opened a new log
I20260925 09:19:26.039247 26182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } }
I20260925 09:19:26.039947 26182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5285199b039f43a2adcf28d86d3bb964, State: Initialized, Role: FOLLOWER
I20260925 09:19:26.040748 26182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } }
I20260925 09:19:26.048448 26293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f: Generated new TSK 0
I20260925 09:19:26.050400 26293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 09:19:26.059262 26157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6f83f9da3b147d18a2fae2896fa978f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5285199b039f43a2adcf28d86d3bb964"
I20260925 09:19:26.059774 26157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964 [term 0 FOLLOWER]: Advancing to term 1
I20260925 09:19:26.063155 26157 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f83f9da3b147d18a2fae2896fa978f in term 1.
I20260925 09:19:26.063484 26182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 09:19:26.065001 26306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } } }
I20260925 09:19:26.065425 26306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:19:26.070636 26156 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964 [term 1 FOLLOWER]: Refusing update from remote peer c6f83f9da3b147d18a2fae2896fa978f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 09:19:26.071708 26289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 09:19:26.129719 26306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6f83f9da3b147d18a2fae2896fa978f. Latest consensus state: current_term: 1 leader_uuid: "c6f83f9da3b147d18a2fae2896fa978f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } } }
I20260925 09:19:26.129693 26312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6f83f9da3b147d18a2fae2896fa978f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } } }
I20260925 09:19:26.130200 26306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 09:19:26.130368 26312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 09:19:26.170359 3149 mini_kdc.cc:348] Time spent kdestroy: real 0.107s user 0.000s sys 0.024s
I20260925 09:19:26.181643 26325 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260925 09:19:26.228531 26186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:26.209808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38810 (local address 127.3.19.125:33195)
0925 09:19:26.210013 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:26.210020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:26.225596 (+ 15576us) server_negotiation.cc:407] Connection header received
0925 09:19:26.226469 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:26.226479 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:26.226596 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:26.226692 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:26.228197 (+ 1505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38810: BlockingRecv error: recv got EOF from 127.0.0.1:38810 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 09:19:26.230942 26110 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:26.210020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.3.19.126:42489)
0925 09:19:26.210532 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:26.210541 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:26.224951 (+ 14410us) server_negotiation.cc:407] Connection header received
0925 09:19:26.229196 (+ 4245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:26.229205 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:26.229340 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:26.229428 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:26.230699 (+ 1271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42666: BlockingRecv error: recv got EOF from 127.0.0.1:42666 (error 108)
Metrics: {"server-negotiator.queue_time_us":306}
W20260925 09:19:26.231593 26300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:26.219158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.3.19.124:34373)
0925 09:19:26.219414 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:26.219422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:26.229789 (+ 10367us) server_negotiation.cc:407] Connection header received
0925 09:19:26.230026 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:26.230033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:26.230139 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:26.230213 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:26.231345 (+ 1132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55256: BlockingRecv error: recv got EOF from 127.0.0.1:55256 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 09:19:26.238636 26186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:26.236785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.3.19.125:33195)
0925 09:19:26.236954 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:26.236961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:26.237148 (+ 187us) server_negotiation.cc:407] Connection header received
0925 09:19:26.237375 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:26.237381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:26.237508 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:26.237630 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:26.238348 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38816: BlockingRecv error: recv got EOF from 127.0.0.1:38816 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 09:19:26.251572 26186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:26.243178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.3.19.125:33195)
0925 09:19:26.243333 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:26.243339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:26.249814 (+ 6475us) server_negotiation.cc:407] Connection header received
0925 09:19:26.250294 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:26.250300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:26.250390 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:26.250506 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:26.251354 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38824: BlockingRecv error: recv got EOF from 127.0.0.1:38824 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 09:19:26.258620 26186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:26.256584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.3.19.125:33195)
0925 09:19:26.256741 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:26.256747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:26.257216 (+ 469us) server_negotiation.cc:407] Connection header received
0925 09:19:26.257400 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:26.257405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:26.257513 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:26.257623 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:26.258386 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38840: BlockingRecv error: recv got EOF from 127.0.0.1:38840 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 09:19:26.268541 26186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:26.266458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38856 (local address 127.3.19.125:33195)
0925 09:19:26.266660 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:26.266666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:26.267167 (+ 501us) server_negotiation.cc:407] Connection header received
0925 09:19:26.267349 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:26.267355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:26.267450 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:26.267552 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:26.268307 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38856: BlockingRecv error: recv got EOF from 127.0.0.1:38856 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260925 09:19:26.282117 26186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:26.279948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.3.19.125:33195)
0925 09:19:26.280116 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:26.280123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:26.280623 (+ 500us) server_negotiation.cc:407] Connection header received
0925 09:19:26.280862 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:26.280868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:26.280965 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:26.281069 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:26.281885 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38862: BlockingRecv error: recv got EOF from 127.0.0.1:38862 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 09:19:26.304783 26186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:26.302134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38870 (local address 127.3.19.125:33195)
0925 09:19:26.302329 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:26.302336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:26.302857 (+ 521us) server_negotiation.cc:407] Connection header received
0925 09:19:26.303185 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:26.303192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:26.303293 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:26.303388 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:26.304538 (+ 1150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38870: BlockingRecv error: recv got EOF from 127.0.0.1:38870 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 09:19:26.343199 26186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:26.340684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.3.19.125:33195)
0925 09:19:26.340878 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:26.340884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:26.341488 (+ 604us) server_negotiation.cc:407] Connection header received
0925 09:19:26.341710 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:26.341717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:26.341831 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:26.341981 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:26.342911 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38884: BlockingRecv error: recv got EOF from 127.0.0.1:38884 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 09:19:26.414268 26186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:26.412512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.3.19.125:33195)
0925 09:19:26.412723 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:26.412729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:26.412980 (+ 251us) server_negotiation.cc:407] Connection header received
0925 09:19:26.413159 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:26.413164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:26.413271 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:26.413376 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:26.414059 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38894: BlockingRecv error: recv got EOF from 127.0.0.1:38894 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 09:19:26.548686 26186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:26.546390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.3.19.125:33195)
0925 09:19:26.546561 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:26.546566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:26.547102 (+ 536us) server_negotiation.cc:407] Connection header received
0925 09:19:26.547332 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:26.547337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:26.547414 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:26.547521 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:26.548414 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38900: BlockingRecv error: recv got EOF from 127.0.0.1:38900 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 09:19:26.692454 26377 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6f83f9da3b147d18a2fae2896fa978f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f83f9da3b147d18a2fae2896fa978f" member_type: VOTER last_known_addr { host: "127.3.19.126" port: 42489 } } peers { permanent_uuid: "5285199b039f43a2adcf28d86d3bb964" member_type: VOTER last_known_addr { host: "127.3.19.125" port: 33195 } } peers { permanent_uuid: "9e64f08e26e44935a398c82c3c9de672" member_type: VOTER last_known_addr { host: "127.3.19.124" port: 34373 } } }
I20260925 09:19:26.692895 26377 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f83f9da3b147d18a2fae2896fa978f [sys.catalog]: This master's current role is: LEADER
I20260925 09:19:26.781581 26273 catalog_manager.cc:1259] Loaded cluster ID: 512a606894234861ba761184caea90ec
I20260925 09:19:26.781682 26273 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672: loading cluster ID for follower catalog manager: success
I20260925 09:19:26.784135 26273 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672: acquiring CA information for follower catalog manager: success
I20260925 09:19:26.785933 26273 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e64f08e26e44935a398c82c3c9de672: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 09:19:26.810361 26186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 09:19:26.808489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38902 (local address 127.3.19.125:33195)
0925 09:19:26.808728 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:26.808735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:26.808920 (+ 185us) server_negotiation.cc:407] Connection header received
0925 09:19:26.809160 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:26.809164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:26.809295 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:26.809406 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 09:19:26.810153 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38902: BlockingRecv error: recv got EOF from 127.0.0.1:38902 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 09:19:27.185999 26325 catalog_manager.cc:1259] Loaded cluster ID: 512a606894234861ba761184caea90ec
I20260925 09:19:27.186161 26325 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964: loading cluster ID for follower catalog manager: success
I20260925 09:19:27.188773 26325 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964: acquiring CA information for follower catalog manager: success
I20260925 09:19:27.190768 26325 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5285199b039f43a2adcf28d86d3bb964: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 09:19:27.336054 26392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:27.327919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.3.19.125:33195)
0925 09:19:27.328643 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:27.328650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:27.328967 (+ 317us) server_negotiation.cc:407] Connection header received
0925 09:19:27.329183 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:27.329187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:27.329417 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:27.329561 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:19:27.330135 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:27.331015 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:27.333140 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:27.333842 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:27.333958 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:27.334630 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:19:27.335375 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:19:27.335501 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:27.335650 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 09:19:27.335730 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":343,"threads_started":1}
I20260925 09:19:27.350646 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:27.343122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.3.19.124:34373)
0925 09:19:27.343946 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:27.343952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:27.343971 (+ 19us) server_negotiation.cc:407] Connection header received
0925 09:19:27.344087 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:27.344091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:27.344247 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:27.344407 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:19:27.344941 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:27.345860 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:27.348047 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:27.348494 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:27.348614 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:27.349389 (+ 775us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:19:27.349927 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:19:27.350071 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:27.350225 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 09:19:27.350309 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":599,"threads_started":1}
I20260925 09:19:27.375043 26392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:27.357471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38926 (local address 127.3.19.125:33195)
0925 09:19:27.357870 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:27.357879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:27.367769 (+ 9890us) server_negotiation.cc:407] Connection header received
0925 09:19:27.368098 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:27.368107 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:27.368224 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:27.368308 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:19:27.369107 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:27.370393 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:27.372685 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:27.373425 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:27.373555 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:27.374172 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:19:27.374300 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:19:27.374373 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:27.374634 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 09:19:27.374729 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260925 09:19:27.375609 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:27.358282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.3.19.126:42489)
0925 09:19:27.366141 (+ 7859us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:27.366149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:27.366179 (+ 30us) server_negotiation.cc:407] Connection header received
0925 09:19:27.366335 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:27.366343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:27.366623 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:27.366859 (+ 236us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:19:27.368902 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:27.370089 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:27.372426 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:27.373083 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:27.373806 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:27.374020 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:19:27.374508 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:19:27.375158 (+ 650us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:27.375214 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 09:19:27.375306 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7683,"thread_start_us":7554,"threads_started":1}
I20260925 09:19:27.375967 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 09:19:27.358049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.3.19.124:34373)
0925 09:19:27.358199 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 09:19:27.358205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 09:19:27.368683 (+ 10478us) server_negotiation.cc:407] Connection header received
0925 09:19:27.368819 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 09:19:27.368825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 09:19:27.368912 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 09:19:27.369006 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 09:19:27.369916 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:27.371097 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:27.373595 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 09:19:27.374238 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 09:19:27.374331 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 09:19:27.374958 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 09:19:27.375096 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 09:19:27.375174 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 09:19:27.375597 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 09:19:27.375685 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 09:19:27.380591 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 26041
I20260925 09:19:27.381659 26103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:19:27.496686 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 26041
I20260925 09:19:27.528210 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 26115
I20260925 09:19:27.529623 26177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:19:27.646283 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 26115
I20260925 09:19:27.670146 3149 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu pid 26189
I20260925 09:19:27.671336 26251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 09:19:27.778661 3149 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCZfsc6/build/asan/bin/kudu with pid 26189
2026-09-25T09:19:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4970 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10060 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (367284 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST